The Semiotic Engineering of Human-Computer

Interaction
Acting with Technology
Bonnie Nardi, Victor Kaptelinin, and Kirsten Foot, editors
Tracing Genres through Organizations: A Sociocultural Approach to
Information Design
Clay Spinuzzi, 2003
Activity-Centered Design: An Ecological Approach to Designing Smart Tools
and Usable Systems
Geri Gay and Helene Hembrooke, 2004
The Semiotic Engineering of Human-Computer Interaction
Clarisse Sieckenius de Souza, 2004
The Semiotic Engineering of Human-Computer
Interaction
Clarisse Sieckenius de Souza
The MIT Press
Cambridge, Massachusetts
London, England
2005 Massachusetts Institute of Technology
All rights reserved. No part of this book may be reproduced in any form by any electronic
or mechanical means (including photocopying, recording, or information storage and
retrieval) without permission in writing from the publisher.
MIT Press books may be purchased at special quantity discounts for business or sales pro-
motional use. For information, please email [email protected] or write to
Special Sales Department, The MIT Press, 5 Cambridge Center, Cambridge, MA 02142.
This book was set in Sabon by SNP Best-set Typesetter Ltd., Hong Kong
Printed and bound in the United States of America.
Library of Congress Cataloging-in-Publication Data
De Souza, Clarisse Sieckenius.
The semiotic engineering of human-computer interaction / Clarisse Sieckenius de Souza.
p. cm. (Acting with technology)
Includes bibliographical references and index.
ISBN 0-262-04220-7 (alk. paper)
1. Human-computer interaction. 2. Semiotics. I. Title. II. Series.
QA76.9.H85D465 2005
004.019dc22
2004055937
10 9 8 7 6 5 4 3 2 1
For Lucas, Karina, Magnus, and Biba
Contents
Series Foreword ix
List of Illustrations xi
List of Tables xv
List of Abbreviations xvii
Preface xix
I Foundation 1
1 Introduction 3
2 Fundamental Concepts in Semiotics 35
3 Semiotic Engineering 83
II Derivation 109
4 Communicability and the Designers Deputy Discourse 111
5 The Semiotic Engineering of Customizable and Extensible Applications 163
6 The Semiotic Engineering of Multi-User Computer Applications 201
III Questions 245
7 Reection 247
Notes 263
Bibliographic References 265
Software and Online Databases and Services 273
Author Index 275
Subject Index 279
Software Index 285
Series Foreword
The MIT Press Acting with Technology series is concerned with the study of mean-
ingful human activity as it is mediated by tools and technologies. The goal of the
series is to publish the best new booksboth research monographs and textbooks
that contribute to an understanding of technology as a crucial facet of human activ-
ity enacted in rich social and physical contexts.
The focus of the series is on tool-mediated processes of working, playing, and
learning in and across a wide variety of social settings. The series explores devel-
opments in post-cognitivist theory and practice from the elds of sociology, com-
munication, education, and organizational studies, as well as from science and
technology studies, human-computer interaction, and computer-supported collabo-
rative work. It aims to encompass theoretical frameworks, including cultural-
historical activity theory, actor network theory, distributed cognition, and those
developed through ethnomethodological and grounded theory approaches.
This book deals with communication, signs, and sense making in human-
computer interaction. Its main focus is on a basic aspect of design and use of infor-
mation technologies, which is often underestimated or altogether ignored. De Souza
takes the perspective that interactive technologies are metacommunication artifacts.
Not only do they enable and support interaction, they also convey to users design-
ers visions of communication with and through artifacts.
Designers, de Souza claims, are in a very real sense present in the interaction
between users and technologies. Their visions direct, constrain, and in other ways
determine users understanding of the system in general and user interface features
in particular. Metacommunication messages sent by designers are gradually, and not
necessarily accurately, discovered and interpreted by users. The book proposes an
approach intended to help address specic problems of designing information tech-
nologies as metacommunication artifacts. The approach, semiotic engineering,
draws upon a rich corpus of semiotic theories. By applying these theories to human-
computer interaction, de Souza identies a range of design concepts and strategies,
which are illustrated with concrete examples.
The book integrates a wealth of novel, exciting, and useful ideas, which are both
theoretically sound and practically useful. We believe it is valuable reading for
anyone interested in technology studies or semiotics, as well as human-computer
interaction.
Bonnie Nardi, Victor Kaptelinin, and Kirsten Foot
x Series Foreword
Illustrations
Figure 1.1 SmartFTP. 6
Figure 1.2 User-centered design and semiotic engineering. 8
Figure 1.3 WinVI. 11
Figure 1.4 Direct manipulation in Treemap. 13
Figure 1.5 Acrobat. 14
Figure 1.6 Notes and highlights in Acrobat documents. 16
Figure 1.7 Menu choices in Acrobat. 17
Figure 1.8 Textual discontinuity of visually continuous text. 18
Figure 1.9 Comments and markups. 19
Figure 1.10 Communicating strategies to a user. 23
Figure 2.1 The Peircean sign. 41
Figure 2.2 Abductive and deductive reasoning. 43
Figure 2.3 Microsoft Excel. 45
Figure 2.4 Most populated states in 1981. 50
Figure 2.5 Most populated states in 1992. 51
Figure 2.6.a Direct object manipulation. 53
Figure 2.6.b Direct concept manipulation. 54
Figure 2.7 Reective direct concept manipulation. 55
Figure 2.8 Screening amed email. 64
Figure 2.9 Jakobsons model. 66
Figure 2.10.a Expressive function. 67
Figure 2.10.b Conative function. 67
Figure 2.10.c Referential function. 68
Figure 2.10.d Phatic function. 69
Figure 2.10.e Metalinguistic function. 70
Figure 2.10.f Poetic function. 70
Figure 2.11 Signs and semiosis in CD Player. 76
Figure 2.12 CD Player buttons. 78
Figure 2.13 Metonymies in Microsoft Word. 81
Figure 2.14 Inconsistencies with metonymical expressions. 82
Figure 3.1 Metacommunication in Arachnophilia. 86
Figure 3.2 The HCI design space. 88
Figure 3.3 The designers deputy discourse. 92
Figure 3.4 Traces of many speakers talking. 94
Figure 3.5 Programming playlists. 102
Figure 3.6 Epistemic tools and knowledge activities. 107
Figure 4.1 Composing messages with Eudora. 116
Figure 4.2.a An attempt to switch signatures. 117
Figure 4.2.b Looking for an option in the Special menu. 118
Figure 4.2.c Another frustrated attempt. 118
Figure 4.2.d Exploring Options. 119
Figure 4.2.e Pasting signature at the end of message. 120
Figure 4.2.f How to switch signatures in Eudora. 121
Figure 4.3 Recorded interaction and communicability utterances. 127
Figure 4.4 Types of information required in task and interaction models. 156
Figure 4.5 An online help system. 159
Figure 5.1 Different signication territories. 165
Figure 5.2 Changing computer language dimensions. 167
Figure 5.3 Semiotic aspects of computer language dimensions. 173
Figure 5.4 Communicative dimensions of signication codes. 176
Figure 5.5 Effects of recursive macro application. 177
Figure 5.6 DOS window and command. 179
Figure 5.7 Semiotic dimensions for customization and extensions. 181
Figure 5.8 Semiotic correspondences between human and computer
perspectives. 182
xii List of Illustrations
Figure 5.9 StageCast
TM
Creators screen shot. 184
Figure 5.10 Microsofts interface agent. 188
Figure 5.11 Alternative interface patterns for the same interactive
pattern. 192
Figure 6.1.a The system as a communication center. 202
Figure 6.1.b The system as a virtual environment for interaction. 202
Figure 6.1.c The system as a telecommunications device. 202
Figure 6.2 The communication center metaphor in MSN Messenger. 203
Figure 6.3 Microsoft Assistant helps Microsoft Outlook user. 204
Figure 6.4 The virtual environment metaphor in Active Worlds. 206
Figure 6.5 PhoneWorks. 208
Figure 6.6 Virtual address book. 209
Figure 6.7 A semiotic inspection of MUApps. 218
Figure 6.8 MArq-G* components. 236
List of Illustrations xiii
Tables
Table 4.1 Categorization of communicability utterances. 138
Table 4.2 Pairing cognitive walkthrough and communicability evaluation. 142
Table 4.3 Pairing heuristic evaluation and communicability evaluation. 144
Table 4.4 Sources of metacommunication contents. 158
List of Abbreviations
AI articial intelligence
CMC computer-mediated communication
CSCW computer-supported collaborative work
DCM direct concept manipulation
DOM direct object manipulation
EUD end-user development
EUP end-user programming
GUI graphical user interface
HCI human-computer interaction
LAP language-action perspective
NLP natural language processing
RDCM reective direct concept manipulation
TSP theory of sign production
UCD user-centered design
WYSIWYG what you see is what you get
Preface
Back in the late eighties when I nished my Ph.D. in linguistics, I began to teach
graduate courses in natural language processing (NLP) in a computer science depart-
ment. As a visiting professor without any formal training in computing, I used the
tools I had learned from formal linguistics to build my own understanding of com-
puters and computing. I was then struggling to learn enough about my students
backgrounds, so that I could share my vision of NLP with them before I started to
talk about scientic and technical issues. Because my students and I came from very
different academic traditions, we explored all possible connections in what we knew.
There was a lot of intellectual excitement in nding analogies and contrasts, and
this is how this book started. Of course I was not aware of what was really hap-
pening then, just as I may probably not be aware of what is really happening
now.
After some years teaching NLP and other topics in articial intelligence, I learned
to cross-examine theories springing from different disciplines and started to nd my
way in computer science. I realized that I had been using a kind of intuitive semi-
otic analysis to interpret a eld that was new to me and to teach concepts from a
new eld to students trained in computer science. So I decided to explore semiotics
methodically. I thought that this way of teaching and learning could be used to build
not only scholarly accounts of theories and relations among them, but also very
nave accounts of how complicated things work. And this led me into human-
computer interaction (HCI), and into applying semiotics to try to tell users what
computer programs mean and how they can be used.
This book is about sense making and how HCI can be designed not only to help
users understand computers, but in fact to help them come closer to understanding
computing. Computing is playing with signs. Every program or system is like a
game. You can only play if you can make sense of the rules. And once you learn
the rules, you immediately begin to see how rules can be changed and how the game
can be adapted to many different play situations.
At the heart of every game is communication and how the players agree to rep-
resent and interpret their moves. This is the essence of semiotic engineeringa
theory of HCI, where communication between designers and users at interaction
time is the focus of investigation. Designers must somehow be there to tell users
how to play with the signs they have invented, so that the game can start. The theory
makes many connections among concepts borrowed from semiotics and computer
science and produces a different account of HCI if compared to existing theories of
cognitive breed. Hopefully it will help designers communicate their vision to users
in much easier ways. And maybe then users will begin to understand designs, and
not just struggle to learn which controls to activate.
I have many people to thank for having come this far. In particular, I am thank-
ful to Terry Winograd for having invited me to be a visiting researcher at the Center
for the Study of Language and Information (CSLI), where I learned the foundations
of HCI and started to shape the thoughts presented in this book. I am also very
thankful to Tom Carey, who patiently listened to the ongoing semiotic engineering
story in its early years and showed me where I could go with it. Don Norman asked
me many difcult questions on different occasions and helped me to see what makes
theories differ from one another. Ben Shneiderman pushed me into thinking about
why and how semiotic engineering is expected to affect the quality of HCI prod-
ucts and, ultimately, the users experience. Ben has also been incredibly supportive
in helping me nd my way in book writing. Jenny Preece generously commented on
many of my papers and on the complete manuscript. She often called my attention
to where and how I should try to make my work more readable. More than a helpful
colleague, Jenny has been a dear friend, encouraging me to keep on trying when
promising roads occasionally turned out to be dead ends.
I have had the privilege of having wonderful students and colleagues at PUC-Rio,
without whom I would have never been able to achieve this project. My doctoral
students, in particular, have made capital contributions to the theory. I am deeply
grateful to my colleagues and friends Raquel Prates and Simone Barbosa, whose
intellectual and emotional support is encoded in many lines of this book.
I would not have come this far without my loving family. Altair, Ingrid, Ib, Cid,
Tania, Mnica, and Cristina have been my guardian angels throughout this project.
Our teamwork for this project in Pittsburgh and Rio de Janeiro is an unforgettable
xx Preface
sign of love, and I am lucky that I can just go on interpreting this sign forever,
in truly unlimited semiosis.
I would nally like to thank CNPqthe Brazilian Council for Scientic and Tech-
nological Developmentfor the nancial support that I have received to do a large
portion of this research. And also Kathy Caruso and the other editors and staff at
The MIT Press, for helping me improve my nonnative writing and make this book
much more readable.
C. S. S.
Rio de Janeiro
December 2003
Preface xxi
I
Foundation
There is interaction if there is a call for it,
no interaction if there is no call for it.
Zen master Yangshan
1
Introduction
Semiotic or semiotics are terms not frequently found in the HCI literature.
Search results can be taken as a rough indication of the relative scarcity of semiotic
approaches to HCI. In December 2003 a search for semiotic in the HCI Bibliogra-
phy (see HCI Bibliography) returned 22 records, whereas a search for ethnographic
returned 104, one for ergonomic returned 344, and nally one for cognitive returned
1,729. Although these numbers cannot be taken as a precise measure, the difference
in scale speaks for itself. Semiotics is the study of signs, signication processes, and
how signs and signication take part in communication. It is neither a new disci-
pline, nor one whose object of investigation is foreign to HCI. In its modern guise,
it has been established for approximately one century. But the debate about its
central themes of investigationmeaning assignment, meaning codication, and the
forms, ways, and effects of meaning in communicationdates back to the Greek
classics.
Most of the relatively rare allusions to semiotic principles or theories in HCI con-
centrate on aspects of graphical user interfaces and visual languages. However, the
World Wide Web and the multicultural issues of Internet applications have raised
another wave of interest in semiotics. Since sign systems are produced and perpet-
uated by cultures, some contemporary semioticians dene semiotics as a theory of
culture (Eco 1976; Danesi and Perron 1999). Their views are naturally an attrac-
tive foundation for those interested in understanding the nature and enhancing the
design of computer-mediated communication in various social contexts, such as
computer-supported collaborative work, online communities, computer-supported
learning, and distance education.
1.1 Semiotic Theories of HCI
To date only a few authors have taken a broader and more radically committed
semiotic perspective on the whole eld of HCI (see, e.g., Nadin 1988; Andersen,
Holmqvist, and Jensen 1993; de Souza 1993; Andersen 1997). This is mainly
because, in spite of its being a resourceful discipline for HCI researchers, teachers,
and practitioners, semiotics covers a vast territory of concepts and uses a wide
variety of analytic methods that are neither familiar nor necessarily useful to them.
Semiotic engineering is one of the few attempts to bring together semiotics and HCI
in a concise and consistent way, so as to support new knowledge organization and
discovery, the establishment of useful research methods for analysis and synthesis,
and also the derivation of theoretically sound tools for professional training and
practice.
First presented as a semiotic approach to user interface design (de Souza 1993),
semiotic engineering has evolved into a theory of HCI. As its name implies, it draws
on semiotics and on engineering to build a comprehensive theoretical account of
HCI. Semiotics is important because HCI involves signication and meaning-related
processes that take place in both computer systems and human minds. And engi-
neering is important because the theory is expected to support the design and
construction of artifacts. Moreover, semiotics and engineering become tightly
coupled when we think that HCI artifacts are intellectual constructs, namely, the
result of choices and decisions guided by reasoning, sense making, and technical
skills, rather than predictable natural laws. Like all other intellectual products, HCI
artifacts are communicated as signs, in a particular kind of discourse that we must
be able to interpret, learn, use, and adapt to various contexts of need and oppor-
tunity. Thus, the semiotic engineering of HCI artifacts is about the principles, the
materials, the processes, the effects, and the possibilities for producing meaningful
interactive computer system discourse.
The semiotic engineering account of HCI makes explicit references to the theorys
object of investigation, the interests and values involved in establishing this object
as such, the epistemological conditions and methodological commitments that affect
the results of investigation, and the prospect of scientic knowledge advancement
in relation to its perceived conditions of use and validity. It also aims to provide the
basis for useful technical knowledge for HCI professionsthat is, knowledge that
can be translated into professional tools (abstract or concrete), knowledge that can
improve the quality of professional products or services, knowledge that can be
4 Chapter 1
taught to young professionals and rened by practice, and knowledge that can be
consistently compared to other practical knowledge, and be complemented, sup-
plemented, or replaced by it.
The importance of dealing explicitly with epistemological issues in a theory of
HCI, namely, with issues related to how knowledge is gained, analyzed, tested, and
used or rejected lies in our need to discriminate the validity, the reach, and the appli-
cability of HCI knowledge coming from such widely different areas as computer
science, psychology, sociology, anthropology, linguistics, semiotics, design, and
engineering, among others. We aim to help semiotic engineering adopters identify
the need and the opportunity to use the knowledge this theory proposes, as well
as the limitations and gaps that call for further investigation and other types of
theories.
One of the prime advantages of a semiotic perspective on HCI is to center a
researchers attention on signs. Signs have a concrete objective stance that is pro-
duced and interpreted by individuals and groups in a variety of psychological, social,
and cultural contexts. They are encoded in natural or articial signication systems
of widely diverse kinds, and they are typically used to communicate attitudes,
intents, and contents in a multiplicity of media. Most semiotic approaches to HCI
view computer programs as (sources of) articial signication systems, and com-
puter-based devices as media. For example, in gure 1.1 we see various signs in
SmartFTP v.1.0.979s interface: English words and technical terms on the main
menu bar; drawings and sketches of various objects on toolbars; pull-down menus,
text boxes, grids, and other widgets. Each one means something, to users and design-
ers. Other signs appear as a result of interaction, such as the ability to drag items
from a SmartFTP window into another applications window or workspace, and
vice versa. These signs also mean things, such as the users convenience or a systems
requirement. SmartFTP incorporates a complex signication system that users must
understand in order to take full advantage of the systems features.
The attitudes, intents, and contents communicated through interactive systems in
computer media are those of the various stakeholders involved in the development
of HCI artifactsclients, owners, users, developers, designers, technical support
professionals. Two of them have a distinguished status in HCI: designers and users.
Designers, because they must be able to embed in the artifact they are about to
produce the whole range of messages that are expected by and useful to all stake-
holders (including themselves). And users, because they are the ultimate judges of
whether the artifact is good or not.
Introduction 5
6 Chapter 1
Figure 1.1
SmartFTP v.1.0.979s interface. Screen shot reprinted by courtesy of SmartFTP
(http://www.smartftp.com).
1.2 The Semiotic Engineering Framework
Semiotic engineering starts from this general semiotic perspective and assigns to both
designers and users the same role in HCI, namely, that of interlocutors in an overall
communicative process. Designers must tell users what they mean by the artifact
they have created, and users are expected to understand and respond to what they
are being told. This kind of communication is achieved through the artifacts inter-
face, by means of numerous messages encoded in words, graphics, behavior, online
help, and explanations. Thus, by using this theory to study, analyze, and make deci-
sions about users and their expected reactions, designers are simultaneously study-
ing, analyzing, and making decisions about their own communicative behavior and
strategies. Semiotic engineering is therefore an eminently reective theory, which
explicitly brings designers onto the stage of HCI processes and assigns them an onto-
logical position as important as that of the users.
This perspective is considerably different from and complementary to the user-
centered perspective that has shaped our eld in the last two decades following
Norman and Drapers seminal publication (1986). In user-centered design (UCD),
designers try to identify as precisely as possible what the users want and need. User
studies and task analysis allow designers to form a design model that matches such
wants and needs. The design model is projected by the system image, which users
must understand and interact with to achieve their goals. Thus the system image is
the ultimate key to success. If the design model is conveyed through the appropri-
ate system image, which rests on familiar concepts and intuitive relations, users can
easily grasp and remember how the system works. In semiotic engineering, however,
although designers also start by trying to understand users and what they need or
want to do, they are not just trying to build the system image. They are trying to
communicate their design vision to users. Their design vision amounts to how users
may or must interact with a particular computer system in order to achieve their
goal through many communicative codes integrated in the systems interface. In
gure 1.2 we see a schematic comparison of UCD and semiotic engineering. On the
UCD side, the system image is the only trace of all the intellectual work that has
taken place during design, and it is what the user is required to learn. On the semi-
otic engineering side, the designer herself is present at interaction time, telling the
user about her design vision, to which the user will respond in various ways (includ-
ing unexpected and creative ways). The ideal of UCD is that the user model cap-
tures the essence of the design model, projected in the system image. The ideal of
Introduction 7
semiotic engineering is that designer and user understand each other, and that users
nd the designers vision useful and enjoyable.
The gist of user-centeredness is a radical commitment to understanding the users
and asking them what they want and need. Their answers set the goals that pro-
fessional HCI design practices must meet for interactive computer-based artifacts to
be usable, useful, and widely adopted. Most of the technical knowledge required
for successful UCD has resulted from empirical studies that attempted to associate
certain features of interactive software to certain kinds of user behavior and reac-
tion. Design features and patterns that have demonstrably met user-centeredness
targets in a statistically signicant quantity of situations have been compiled into
legitimate technical knowledge in the form of principles, heuristic rules, and guide-
lines. The greatest difculty with this kind of research practice in our eld, however,
has been the cost of sound predictive theories, especially in view of the speed of
technological progress. The statistic validity of empirical studies requires that
8 Chapter 1
Figure 1.2
User-centered design compared to semiotic engineering.
numerous experiments be run with hundreds, if not thousands, of subjects before
any piece of knowledge is legitimately added to an existing theory.
But two familiar obstacles have been always in the way of such extensive studies
the pressure exercised by industrial stakeholders, who cant afford the time and
money involved in extensive latitudinal and/or longitudinal user studies, and the
pace of technological change, which often precipitates the obsolescence of systems
and techniques employed in long-term research projects. As a result, sound predic-
tive theories of HCI have typically concentrated on specic phenomena that cut
across technologies (e.g., Fitts Law [Fitt 1954]), but are of little use when it comes
to making other types design decisions (e.g., choosing between adaptive or cus-
tomizable interfaces). Moreover, we have witnessed a certain glorication of guide-
lines and heuristic rules. In some cases, practitioners have focused on conforming
to guidelines and heuristic rules instead of using them appropriately as decision-
support tools. Others have overemphasized the value of checklists and forgone much
of their own intellectual ability to analyze and critique the artifact they are about
to build. Those who exercise this power have often been misled into thinking that
because guidelines dont take them all the way to a nal and unique design solu-
tion, research work from which guidelines are derived is really of little use (Bellotti
et al. 1995; Rogers, Bannon, and Button 1994).
Adding other types of theories to HCI, such as explanatory theories and inter-
vention-oriented theories (Braa and Vidgen 1997), helps us gain sound knowledge
that supports decision making in design, although not on the basis of empirical data.
Conceptual schemas and interpretive models can enrich our analysis of problem sit-
uations and candidate solutions. They can also help us frame problems in different
ways and nd opportunities for exploring design paths that would possibly not
emerge in causal reasoning processes that are typical of predictive theories.
1.3 Theorizing about Software as an Intellectual Artifact
In order to illustrate the need and opportunity for such other types of theories
and research practices (i.e., nonpredictive), let us explore the intellectual nature
of software artifacts. What is an intellectual artifact? What is not an intellectual
artifact?
All artifacts are by denition nonnatural objects created by human beings. Some
of them are concrete, like forks and knivesmaterial artifacts that were made to
facilitate certain eating practices in our culture. Others are more abstract, like safety
Introduction 9
measuresprocedural artifacts that are meant to prevent accidents. Some are meant
for physical purposes, like chairs. Others are meant for mental purposes, like logic
truth tables. Thus, strictly speaking, all artifacts result from human ingenuity and
intellectual exercise. But what we call an intellectual artifact is one that has the
following features:

it encodes a particular understanding or interpretation of a problem situation;

it also encodes a particular set of solutions for the perceived problem situation;

the encoding of both the problem situation and the corresponding solutions is
fundamentally linguistic (i.e., based on a system of symbolsverbal, visual, aural,
or otherthat can be interpreted by consistent semantic rules); and

the artifacts ultimate purpose can only be completely achieved by its users if they
can formulate it within the linguistic system in which the artifact is encoded (i.e.,
users must be able to understand and use a particular linguistic encoding system in
order to explore and effect the solutions enabled through the artifact).
According to this denition, knives and forks are not intellectual artifacts, nor
are safety measures and chairs. But logic truth tables are intellectual artifacts, and
so are books (if we think of their content), although books can also be viewed as
physical artifacts (if we think of the physical media where ideas and information
are stored). All intellectual artifacts require that producer and consumer use the
same language. And language is not a metaphor in this case; it is a genuine system
of symbols with a dened vocabulary, grammar, and set of semantic rules. Natural
language descriptions have an additional set of pragmatic rules, which rene the
way in which semantic rules are applied in communicative contexts. But articial
languages usually dont include such pragmatic rules in their description.
The advent of graphical user interfaces (GUIs) popularized the idea that software
was some kind of tool. Almost every interface began to include tools, toolboxes,
and toolbars, with explicit reference to concrete physical artifacts. Gibsons theo-
rizing on affordances (Gibson 1979) inspired HCI researchers (Norman 1988), and
the idea of direct manipulation (Shneiderman 1983) turned the tool metaphor into
one of the pillars of HCI design. But soon the metaphor started to show some of
its limitations. For example, the Gibsonian notion of affordance could not be
directly applied to the HCI. Designers intuitively began to speak of putting affor-
dances in the interfaces (Norman 1999); evaluation methods spoke of users
missing and declining affordances (de Souza, Prates, and Carey 2000), a sign that
affordances were themselves being used as a metaphor and not as a technical
concept. The invariant feature of all metaphorical uses of the term was the presence
10 Chapter 1
of an intended meaning, of what designers meant to do, or what they meant to say
with specic interface elements. And discussions turned around the occasional mis-
matches between what was meant by designers and users when they referred to
certain signs in the interface. Relevant issues having to do with how the designers
intent was encoded in the interface, how users decoded (interpreted) them, and how
they used them to express their own intent during interaction could not t into the
category of typical concrete artifacts. They referred essentially to linguistic processes
(though not necessarily involving only natural language signs). And the kinds of
tools needed to support these processes are not material (like hammers, screw-
drivers, paintbrushes, which all apply to concrete artifacts that we can manipulate),
but rather immaterial (like illustrations, demonstrations, explanatory conversations,
clarication dialogs, which all refer to intellectual artifacts that we can learn). They
are epistemic tools, or tools that can leverage our use of intellectual artifacts.
Epistemic tools can benet both designers and users, since they are both involved
in communication with or about the same intellectual artifacts. It may be difcult
to see why we need epistemic tools to use a basic text editor like WinVi, shown
in gure 1.3. The concrete tools in this GUI must all be familiar to computer-
literate users, except perhaps for the last three that mean, respectively, ANSI char-
acter set, DOS character set, and hexadecimal mode. So one might expect any
computer-literate user to be able to write the text seen in gure 1.3, to edit it with
the familiar cut, copy, and paste tools, and to open and save les, just as easily as
he or she would use pencils, pens, paper, classiers, and folders. However, unless
the user is introduced to the idea behind WinVi, there is a chance that the power
of this intellectual artifact will never be understood. Unless the user knows the Vi
Introduction 11
Figure 1.3
WinVi, a graphical user interface for VI. Screen shot reprinted by courtesy of Raphael
Molle.
Editor and its command language, the string operations that can be performed with
WinVi, and the situations in which such operations are desirable and needed, he or
she may end up complaining about the barrenness of this piece of software (e.g.,
compared to user-friendly ones that may not have half the string processing power
of WinVi). Epistemic tools, like a useful online help system, with clever examples,
explanations, and suggestions for how to use WinVi, should be provided with the
artifact in order to enable users to take full advantage of this product.
Treemap 3.2 (a visualization tool for hierarchical structures in which color and
size may be used to highlight the attributes of leaf nodes) is another example of why
epistemic tools are needed, just as theories that can help us build them. Visualiza-
tions allow for comparisons among substructures and facilitate the discovery of pat-
terns. All interactions are based on direct manipulation of interface elements, either
pertaining to the visualization itself or to specic controls applicable to it. In gure
1.4 we see data referring to the 2000 presidential elections in the United States.
Lighter-colored rectangles correspond to states with population ranging from over
10 million to nearly 30 million people. The size of rectangles is proportionate to
Gores votes, whereas the brightness of the color is proportionate to Bushs votes.
The pop-up rectangle provides details about the number of electoral votes and each
of the candidates votes in each state (or rectangle). In the slice and dice visuali-
zation mode shown in gure 1.4, the layout represents the states grouped in a
fashion that mimics the geographical layout. The user may choose which label to
see on the rectangles. On the upper right part of the screen is more information
about Florida, the selected state on the left side. On the lower part, in the Filters
tab, are various sliders that can help the user narrow in on specic ranges of values
for the various attributes of the data.
As was the case with WinVi, Treemap may go underestimated and underutilized
if users are not introduced to the power of this data visualization tool. Although
the interactive patterns are not difcult to learn and support manipulations that can
help the user nd aspects of information that might be lost in tabular presentations,
they are not obvious for average users. The intellectual value added to the tool by
its designers is not clear from the signs in the interface. There is no reference to pat-
terns of data, and no hint of when or why one kind of visualization is better than
the other. But of course the designers included alternative visualizations because
some are better than others in certain situations. However, unless the users are told
about the relative advantages of each or get at least some sort of cue, they may miss
the whole idea of the tool.
12 Chapter 1
Intellectual tools deserve an appropriate presentation or introduction. Not only
in operational terms (which is the most popular kind of online help users get from
software manufacturers), but also (and perhaps more interestingly) about the
problem-solving strategies that the designers had in mind when they produced the
software. A detailed example of how users would benet from a more careful
introduction to knowledge associated with the product will help us draw some
conclusions about the intellectual aspects of software production and use that
nonpredictive theories such as semiotic engineering can bring to light and help
explore. To this end I will employ a use scenario from Adobe Acrobat 5.0.5.
Introduction 13
Figure 1.4
Direct manipulation and data visualization possibilities in Treemap 3.2. Screen shot
reprinted by courtesy of HCIL, University of Maryland, College Park (http://www
.cs.umd.edu/hcil/Treemap/).
Acrobat offers to users a set of powerful tools for reviewing documents. Among
the most useful are commenting tools and markup tools. In gure 1.5 we see
a screen shot of Acrobat showing its own help documentation as a PDF le that
can be commented and marked up. On the top of the screen is the usual menu bar
and tool bars. On the left-hand side are the bookmarks, which in this case contain
a detailed summary of the document. And on the right-hand side is a document
page linked to the highlighted entry of the summary (marking up documents with
text markup tools). The note tool is an example of a commenting tool, and the
highlight tool is an example of a markup tool. The analogy with concrete arti-
facts such as notes and highlight markers is emphasized by the visual signs appear-
14 Chapter 1
Figure 1.5
Acrobat 5.0.5 standard interface. Adobe product screen shot reprinted with permission
from Adobe Systems Incorporated.
ing on the toolbars on top of the screen. Thus, users can write notes and highlight
portions of text while they are reviewing documents.
Following the UCD cannons, the system image directly projects the idea that users
can call a documents readers attention to portions of the text and write comments
about them. So, for instance, a reasonable reviewing strategy that most users are
likely to adopt is to highlight parts of the text that need to be reviewed and then
write a note with suggestions and justications or questions. The steps to achieve
this goal are as follows:
1. to select the highlight tool
2. to press down the mouse button at the beginning of the text portion to be
highlighted
3. to drag the mouse to the end of the text portion to be highlighted
4. to release the mouse button
5. to select the note tool
6. to click on the document page close to the highlighted portion
7. to double-click on the note icon
8. to type in comments and suggestions
9. to close the note (optional)
The visual effect of these steps can be seen in gure 1.6. The rst two lines of
the top paragraph on the page are highlighted, and next to them is a note where
the user comments that she never knew highlights were comments. This user has
been using Acrobat for over a year now, and her reviewing strategy has always been
what she calls the highlight-and-add-note strategy. But she learned a few things
about this strategy. For example, the spatial contiguity of the note and the high-
lighted text is of course critical. Otherwise, sentences like This is new to me
cannot be correctly interpreted. This must always refer to an object that is spatially
contiguous to it. Another important lesson is about producing a summary of her
comments, which she often does using the summarize tool (see gure 1.7). This
tool generates a listing of comments made by the user. The comments can be l-
tered and sorted in various ways, but the default situation is to list comments and
markups in sequential order of creation. Therefore, when this user generates a
summary of her document, the content of a note she wrote following steps 6 to 8
comes right next to the text she has highlighted in steps 2 to 4. As a result, she can
still interpret this in the listing as referring to the contents of the two highlighted
sentences. The referent of this is shown in the previous entry of the summary.
Introduction 15
At times, when checking all of her comments, she has tried to extend the high-
lighted portion of text to which comments refer. For instance, suppose that after
doing a lot of reviewing on Acrobats help document, the user highlights the other
two lines of the rst paragraph. Assuming that no other comments and markups
have been made on this page, this becomes the third summary entry, and is listed
after the note contents. As a result, although visually the note contents correctly
refer to the four lines in the rst paragraph on the page seen in gures 1.6 and 1.7,
textually the reference is lost in the summary (see gure 1.8). The note comment is
inserted between the rst and the second two lines of the paragraph, and this seems
to refer only to the part of the text, although it really refers to all of it.
16 Chapter 1
Figure 1.6
Notes and highlights in Acrobat documents. Adobe product screen shot reprinted with
permission from Adobe Systems Incorporated.
The trick in Acrobats interface is that, given how the system image is projected,
the highlight-and-add-note strategy is the most obvious to users. And this may lead
to problems in generating summaries. However, as the Acrobats help documenta-
tion is saying, the highlight-and-add-note strategy is not exactly the design vision
for efcient commenting and reviewing. As can be read in gure 1.5, designers do
believe that users may want to highlight or strike through a section of text, and
. . . add a note . . . to explain [the] reason for the markup. However, the way to do
this, in their view, is not to use the note tool but to double-click [on the highlight
or strike-through] to add a note window. But the system image does not suggest
that this is a possibility, or that highlights and notes are both comments
Introduction 17
Figure 1.7
Menu choices for Comments in Acrobat. Adobe product screen shot reprinted with per-
mission from Adobe Systems Incorporated.
(which is precisely what the user is surprised to learn in the documentation). In the
physical world, these are different things.
If Acrobats default congurations are changed, however, the system image gives
the user a hint that maybe notes and highlights have indeed something in common.
The user can change her preferences and choose to see the sequence number of every
comment. The visual effect of this change can be seen in gure 1.9. Note that both
notes and highlights have comment sequence numbers (this is how this congu-
ration parameter is named in the preference dialog). So, they are both comments.
And, since we can double-click on note icons to open small windows and type in
18 Chapter 1
Figure 1.8
Textual discontinuity of visually continuous highlighted text in comment summary. Adobe
product screen shot reprinted with permission from Adobe Systems Incorporated.
comments, we can also expect to double-click on highlighted texts in order to open
the same kind of windows and type in comments.
The effect of this discovery on our user is much deeper than we may at rst
suspect. In fact, when she realizes that every highlight is a comment, just like a note,
it dawns on her that she does not need to create a note to comment every highlight
in the document. She can write her suggestion, justication, or question directly in
the comment window associated with the highlight. And as she begins to do this,
she realizes that the note window referred to in Acrobats documentation indeed
contains the whole extent of marked-up text. If she chooses the following interac-
tive steps:
Introduction 19
Figure 1.9
Visual similarities between comments and markups as a result of changing preferences.
Adobe product screen shot reprinted with permission from Adobe Systems Incorporated.
1. to select the highlight tool
2. to press down the mouse button at the beginning of the text portion to be
highlighted
3. to drag the mouse to the end of the text portion to be highlighted
4. to release the mouse button
5. to double-click on the highlighted text
6. to type in comments and suggestions
7. to close the note (optional)
the summary of comments will look smaller and more readable like this:
Sequence number: 1
Author: clarisse
Date: 12/27/2003 6:57:22 PM
Type: Highlight
The text markup tools provide several methods for visually
annotating text in a document. You can use these comments
by themselves or in conjunction with other comment types.
For example, you may want to highlight or strike through a
section of text, and then double-click to add a note window
to explain your reason for the markup.
This is new to me. Are markups comments?
and not like what is shown in gure 1.8. The design vision, according to what can
be read in Acrobats online documentation, aligns with the comment-on-highlight
strategy, although the system image projects the highlight-and-add-note strategy
with greater salience. Moreover, spatial contiguity problems disappear in the textual
summary.
Without getting into a bug or feature discussion with respect to this example
(i.e., assuming that similarities in visualizations of notes and highlights is a feature
and not an unanticipated side effect of parameter-setting implementation), the truth
is that operating with comment sequence numbers on and off communicates dif-
ferent meanings in Acrobat. And not all of these values have a logical connection
with numbering comments. In particular, if the user knows that highlights (and other
markup tools) automatically generate a comment, this can make reviewing more
agile and consistent. If she doesnt, not only does this introduce unnecessary redun-
20 Chapter 1
dancies in the reviewing process, but this may also cause loss of information (as
when the reader must interpret a word like this, referring to something other than
what is spatially contiguous to it). Users are simply not likely to guess the best strat-
egy from just looking at Acrobats interface and focusing on the system image.
It is useful now to contrast two design objectivesproducing and introducing an
application that meets certain requirements. If Acrobats designers had been given
the goal to produce an application that allows users to operate in different ways
and customize it to meet their needs, the problem with this interface would be that
the system image does not portray equally well all of the products features. In other
words, the exibility to tailor the application to different users proles is there, but
it is poorly communicated. However, if designers had been given the goal to intro-
duce such application, they wouldnt have even partially met their goal. Acrobats
more sophisticated uses are far from self-evident and do not speak for themselves.
They may not go without an explicit introduction. In particular, tactical and strate-
gic choices that users may or should make to increase their productivity and/or sat-
isfaction cannot be derived from the strict meaning of the tools that they see in the
applications interface. Tools tell them a number of things about Acrobats opera-
tional features, but nothing about how this technology may bring about desirable
changes in work strategies or tactics.
1.4 New Nonpredictive Theories for HCI
Nonpredictive theories of HCI, such as the semiotic theories that have guided the
analyses of the examples above, can help us gain access to some relevant issues. The
rst is to add a new perspective on usability and focus on some factors inuencing
usability that are not totally obvious from design guidelines. For example, if we take
Shneidermans eight golden rules (Shneiderman 1998):
1. strive for consistency
2. enable frequent users to use shortcuts
3. offer informative feedback
4. design dialogs to yield closure
5. offer error prevention and simple error handling
6. permit easy reversal of actions
7. support internal locus of control
8. reduce short-term memory load
Introduction 21
we see that they concur to helping users understand the application, of course, but
they may not be sufcient to help designers communicate efciently the intellectual
value added in software tools. Acrobats interface (with respect to our example) can
be said to follow the eight golden rules, but yet it fails to tell the users about some
relevant features of this technology regarding the various reviewing strategies
that users can adopt. So, what do we mean by telling users about the design
vision?
By shifting the design goal statement from producing to introducing technology,
we can approach the answer. Introducing technology aims at making adopters
understand valuable strategic aspects of it, and not only learning how to operate
the system. Users must be told how technology can add value to their work and
activities. Note that the eight golden rules refer to operational aspects: to what kinds
of actions the user must perform, and the resources he needs in order to interact
smoothly with the system. They do not refer to the strategic aspects of the tech-
nology, as for example to the relative advantages of choosing one interactive path
over other possible ones. In the Acrobat example we see that users can easily achieve
their reviewing goals using the highlight-and-add-note strategy (which can be said
to comply with Shneidermans golden rules). But they would very probably be much
more efcient (and would prefer to be so) if they were aware of the comment-on-
highlight strategy that can be adopted by changing preference parameters.
One could argue that changing the label of the show comment sequence
number parameter to something like show comment icons and numbers might
improve the interface and evoke the strategic value of this alternative mode of oper-
ation. But this line of solution is not good enough, for two reasons. One is that it
is episodic and ad hoc, a kind of solution that depends on the designers capacity
to foresee when and why it should be extended to other portions of the interface.
In other words, by keeping with the goal of producing usable interfaces, and using
operational usability metrics to assess the quality of the interface, a designer is not
prompted to think systematically about the strategic decisions of the users. The other
reason is that only by replacing the overall HCI design goal with one of introduc-
ing technology, strategic aspects naturally come rst. If one cannot see why one
should learn a new technology, one can easily choose not to learn it at all. Or, as
seems to be the case with Acrobat, if one doesnt see that certain operations open
the avenue for alternative strategies that can be considerably more efcient than
others, the value of the technology is not fully understood.
22 Chapter 1
So, by shifting focus in stating design goals, we are simultaneously shifting com-
munication priorities in the interface. Strategies come rst, then operations. This
does not necessarily mean that we must have a two-stage communication structure
in the interface, rst telling users everything about the strategies and then, depend-
ing on their choice, telling them all about operations. Neither does it mean that
interfaces must be verbose and loaded with text and explanations. Semiotics can be
used to explore how we can convey both using one and the same sign structure.
Just as an example of how strategies can be communicated along with operations,
consider the proposed redesign of Acrobats Tools menu structure, shown in gure
1.10.
Communicating strategies of use is an important factor in making users under-
stand the design vision. The kind of communication suggested in gure 1.10 would
benet other systems interfaces such as Treemaps and WinVis, illustrated earlier
in this chapter.
1.5 The Contribution of Semiotic Engineering
What effects does a semiotic perspective bring about for HCI research and practice,
and what is the value of semiotic theories in this discipline? To answer these ques-
tions we must analyze how semiotic engineering relates to the other theoriesis it
detrimental to others? is it prone to being complemented or supplemented by others?
In the following subsections we will discuss
Introduction 23
Figure 1.10
Communicating strategies to a usera redesign of Acrobats original menu structure.

the semiotic engineering homogeneous model for users and designers activities
in HCI;

its ontological, epistemological, and methodological constraints and commit-

ments; and

the consequences of having homogeneous models.

1.5.1 A Homogeneous Model for Users and Designers Activities in HCI
Semiotic engineering views HCI as a particular case of computer-mediated human
interaction. This perspective is invariant across most semiotic approaches to HCI
(e.g., Kammersgaard 1988; Nadin 1988; Andersen 1997), but the specicity of semi-
otic engineering is to characterize interactive computer applications as metacom-
munication artifacts (de Souza 1993; Leite 1998). Metacommunication artifacts
communicate a message about communication itself. The message is elaborated and
composed by designers and intended for the users. This is the rst stance in which
HCI designers and users are brought together under the same communicative
process. In order to decode and interpret the designers message, users proceed to
communicate with the message, which gradually unfolds to them all the meanings
encoded in it by the designers.
1
The message actually speaks for the designers in the
sense that it contains all the meanings and supports all meaning manipulations that
the designers have rationally chosen to incorporate in the application in order to
have it do what it has been designed to do. In semiotic engineering terms, the
message serves as the designers deputy, presenting not only an artifact that can
perform a certain range of functions and be used within a certain range of contexts,
but also the rationale and design principles that have been followed while synthe-
sizing this product. Rational decisions and choices relative to meaning and meaning
manipulation are the best evidence that interactive computer applications are intel-
lectual in nature. And as is the case of any other intellectual object its meaning can
only be fully grasped and understood when conveyed in an appropriate semiotic
discourseone in which basic metalinguistic functions are available for mutual
communication.
The four-stage evolutionary schema of metacommunication involving designers
and users in HCI involves the following steps:

The designer studies users, their activities, and their environment;

The designer expresses, in the form of computer technology, his views about how
the users, their activities and environment may or must change (because users so
wish);
24 Chapter 1

Users unfold the designers message through interacting with the system; and

Users nally make full sense of the designers message and respond to it
accordingly.
The whole process thus starts with the designer studying the users, their activity,
and their environment. He takes into account ergonomic, cognitive, social, and cul-
tural factors that can affect usability and adoption of the technology being built.
He consolidates all this knowledge into his interpretation of who the users are, what
they need, what they like and prefer, and so on. The second stage in the process is
an articulation of the knowledge gained in the rst stage with the technical knowl-
edge and creativity of the designer, who then produces an artifact that tells his mind
in the form of an interactive message. The third stage corresponds to the users
unfolding of the designers message, gradually making sense of the various mean-
ings encoded in the artifact. The fourth and last stage in this particular type of
computer-mediated human communication corresponds to the users nal elabora-
tion of the meanings unfolded during interaction, a stage when the user nally makes
full sense of the message. At this stage the users response to the technology is mature
and complete. The user knows the technology, no matter if he is fully satised with
it or not. Semiotic engineering focuses on optimal communication, not on other
crucial aspects of design such as aesthetics, productivity, and user satisfaction, to
name but a few.
The metacommunication artifact is thus a designer-to-user message, whose
meaning can be roughly paraphrased in the following textual schema:
Here is my understanding of who you are, what Ive learned you want or need to do, in
which preferred ways, and why. This is the system that I have therefore designed for you,
and this is the way you can or should use it in order to fulll a range of purposes that fall
within this vision.
Notice that this overall content of the designers message to users holds true in
single-user and multi-user applications, stationary and mobile applications, xed
and extensible applications, Web-based or non-Web-based applications, basically
visual or basically textual applications, virtual reality, work and fun applications,
and so on. It is a robust characterization of HCI across media, domains, and tech-
nology, which is a promising feature for a prospect foundational theory.
Introduction 25
1.5.2 Ontological, Epistemological, and Methodological Constraints and
Commitments
By underwriting a semiotic approach, the semiotic engineering of humancomputer
interaction is bound to live with certain ontological, epistemological and method-
ological constraints and commitments that may or may not exist in other alterna-
tive theories. We should start this discussion by committing to a denition of
semiotics itself. Our choice is to adopt Ecos denition (Eco 1976, 1984), accord-
ing to which semiotics is the discipline devoted to investigating signication and
communication. Signication is the process through which certain systems of signs
are established by virtue of social and cultural conventions adopted by the users,
who are the interpreters and producers of such signs. Communication is the process
through which, for a variety of purposes, sign producers (i.e., signication system
users in this specic role) choose to express intended meanings by exploring the
possibilities of existing signication systems or, occasionally, by resorting to non-
systematized signs, which they invent or use in unpredicted ways.
Ecos denition of semiotics can easily justify why HCI is legitimately an object
for semiotic investigation. It also supports the prevalent view in semiotic approaches
to HCI, namely, that HCI is a particular case of computer-mediated human com-
munication. All interactive computer-based artifacts are meant to express a design
vision according to an engineered signication system. And it nally reveals the
nuances of the semiotic engineering notion of metacommunication artifacts by
helping us see that the signication system used by designers to communicate their
vision to users must become instrumental for the users to express their particular
intent and content back to the application (lest interaction becomes impossible).
But Eco, as many other leading semioticians, fully embraces the fundamental
notion of unlimited semiosis, proposed by the American nineteenth-century philoso-
pher and co-founder of contemporary semiotics, Charles Sanders Peirce. This notion
rests on the idea that a sign is not just a relational structurebinding together a
representation, an object that warrants it the status of being a representation, and
a mental interpretation triggered by it. It is a recursive function which can be applied
to itself any number of times, yielding interpretations that become the argument
of the next function application. The recursive process of its application is called
unlimited semiosis.
In spite of its apparently convoluted formulation, the meaning of unlimited semi-
osis is relatively easy to grasp. In practice, it amounts to a number of interesting
formulations of intuitive facts involved in human communication. It says that the
26 Chapter 1
meaning of a representation cannot be dened as a xed object or class of objects,
although, for there being a representation at all, there must be an object or class of
objects that requires a representation. For example, for the word HCI to be a rep-
resentation (at all) there must be an object or class of objects that directly deter-
mine its codication into a signication system. In the absence of these, nothing
can be a representation of anything. The existence of a mental stance directly
determined by the presence of a representation, but only indirectly determined by
the presence of an object or class of objects, opens the possibility of subjective
(internal) ingredients to interplay with objective (external) ones, generating arbi-
trary interpretations for any given representation. So, there are three components
to a sign: representation, object, and interpretation (often called meaning). For
mutual understanding to be possible, there must be (a) regulatory mechanisms that
can determine if and when shared meanings have been achieved, and (b) generative
mechanisms that can produce other mental stances of meaning when the rst triadic
conguration of semiosis proves to be insufcient for achieving shared meanings.
The principle of unlimited semiosis states that in communication, for instance, semi-
osis will continue for as long as communicating parties are actively engaged in
mutual understanding. Because there is no way to predict the length of this process,
and because in theory it can last for as long as one cares to carry it on, semiosis is
unlimited (although it is nite on an individual scale, given the nitude of our exis-
tence). The same can occur in noncommunicational semiosis, when the human mind
ponders reectively over the meaning of things. This is the very nature of philo-
sophical activity, and by denition it can extend to any length of time, even beyond
the limits of an individuals life span, if we consider schools of thought to behave
as a collective mind. Culture, ideology, archetypal myths are yet other instances of
unlimited semiosis in action.
So, a semiotic theory of HCI, such as the one proposed by semiotic engineering,
has to incorporate signication, communication, and unlimited semiosis in its ontol-
ogy, no matter how uncomfortable the consequences of this move in epistemologi-
cal and methodological terms.
At the ontological level, semiotic engineering splits meaning into two very diverse
categories. Human meanings (that of designers and users) are produced and inter-
preted in accordance with the unlimited semiosis principle. Computer meanings
(human meanings encoded in programs), however, cannot be so produced and inter-
preted. For a computer program, the meaning of a symbol (a kind of degraded
sign, in this context) is no more than an object, or class of objects, encoded into
Introduction 27
another symbol system (or directly into a physical state of a physical device). The-
oretical constraints on algorithmic computation cannot support the notion of unlim-
ited semiosis (Nake and Grabowski 2001) and thus set the borderline between
human and computer semiotics (see section 7.1 for an extensive discussion of this
issue). Moreover, computer interpretation is itself a sign of human interpretations
about meaning and meaning-related processes. The way programs assign meanings
to symbols is a reection, or representation, of how programmers believe meanings
are logically assigned to symbols.
At the epistemological level, some important constraints must be spelled out. First,
in a semiotic perspective, there is no room for a purely objective and stable account
of meaning, whether the designers or the users. Meaning carries inherent subjec-
tive and evolutionary ingredients determined by unlimited semiosis that cast a
shadow of doubt upon the idea that the users context, requirements, and capaci-
ties can be fully captured by any human interpreter at any given time. Even
when we let go of the idea of capturing all meanings, the choice of which are the
relevant ones is equally prone to misjudgment and incompleteness. Consequently,
in this sort of epistemological context, a researcher cannot possibly assume a
positivist attitude, commonly adopted by many who aim to build predictive theo-
ries. From a semiotic perspective one cannot observe and interpret empirical phe-
nomena without being affected by his or her own subjectivity, and the sociocultural
context around him or her. Therefore, the value of observation and interpretation
cannot be dissociated from the purposes the empirical study is designed to serve in
the rst place.
In order to preserve the theory from being declared anarchic and ultimately
useless, for scientic purposes in general and for HCI design in particular, this epis-
temological challenge must be counterbalanced by sound methodological choices.
They must enable researchers to identify the limits of subjectivity, the power of cul-
tural determination, the conditions of social contracts in communicative phenom-
ena, and, in the particular domain of HCI, the effects of computer technology upon
human signication and communication processes. Sound choices will protect
researchers and professional practitioners against adopting a nave idealized view
in which interactive computer-based artifacts can be built according to laws that
are similar to the ones that allow civil engineers to build bridges. And they will also
keep them from adopting a nihilist view in that, because users can never be fully
understood and systems must always be built, any approximation of the actual use
situation is as good as any other. One of the peculiar aspects of this theory is that
28 Chapter 1
it carries in itself a certain ethics of design, which other theories dont explicitly
discuss or assume to exist.
In sum, semiotic engineering operates on a homogeneous continuum of analysis,
where designers, users, and computer artifacts have a role in an overarching com-
municative process. The unit of analysis is a metacommunication artifactan inter-
active computer-based system engineered to communicate to its users the rationale
behind it and the interactive principles through which the artifact can itself be used
to help users achieve a certain range of goals. However, the adoption of a semiotic
ontology of HCI calls for strict epistemological and methodological vigilance. We
believe that this situation pays off for two main reasons. One is that the range of
phenomena encompassed by a semiotic perspective is considerably wider than that
covered by the main theoretical alternatives backing up HCI today. It sufces to
mention, as a justication, that none of them brings together designers, computer
artifacts and users under the same observable unit of investigation. The other is that
we cannot be sure, given the lack of explicit statements about epistemological and
methodological commitments in research work produced by alternative theories,
that similar challenges would not be faced by cognitive, ethnographic, psychosocial,
ergonomic, or purely computational traditions in HCI research. Therefore, it may
well be the case that a stricter vigilance of this sort may yield novel and important
knowledge about the nature of HCI, with positive impacts for both research and
professional practice.
1.5.3 The Consequences of Having Homogeneous Models
A homogeneous continuum of analysis in HCI brings about a certain number of
consequences. First, as we have already mentioned, because users and designers are
involved in the same communicative phenomenon, semiotic engineering becomes a
reective theory. In other words, if designers use semiotic engineering to build inter-
active applications, they must think about their own role and intent in HCI, and
about how their role and intent affect and are affected by those of the users. The
theory is well-suited for problem situations in which introducing interactive tech-
nology seems more important or more precisely the case than producing it.
Second, this homogeneity brings about different possibilities not only for inter-
preting design problems, but also for structuring the solution space and exploring
the comparative value of alternative solutions. These possibilities stem from our
knowledge about signication and communication phenomena, and their rela-
tionship with culture and computer technology. As a consequence, a number of
Introduction 29
constraints and concerns stand out in the design process. For example, when HCI
design is cast as signication system engineering, we realize that this system must
be used in three different situations. To begin, in order to communicate with users
via the designers deputy, designers themselves must use it. This is the context of the
one-shot unidirectional communication voiced by the interactive discourse possi-
bilities programmed into the application. It must also be used by the designers
deputy (or system), which will concretely interact with the user, systematically
generating and interpreting signs according to the semiotic rules embedded in the
program. And nally it must be learned and used by the users, who can only come
to understand, apply, and enjoy a computer application if its signication system
can effectively support communication.
Third, design intent is brought forth as a rst-class citizen in this scenario. Because
designers are actually communicating to users, the purpose of communication must
be achieved. Two consequences follow from this. One is that designers must have
a clear idea of what they want to do and why. The other is that they must be able
to tell this to users in ways that are computable, easy to learn and remember, ef-
cient and effective. Note that they are not talking about telling the users which
button to press to activate this and that function, or what is the meaning of this
and that visual form on the screen. They are telling them why the application makes
sense to them, and why they expect that it will make sense to users, too. One of
the leading reasons why designers expect that users will be pleased with the appli-
cation, and make an effort to learn the specics of it, is that they know who users
are and have designed the application especially for them. If designers cant tell
thisthat is, if they dont know who the users are and have designed an applica-
tion that has some inherent qualities that they expect will be intuitively perceived
and naturally approvedthey must still acknowledge that these qualities are there
and help the users appropriate such qualities.
Fourth, a homogeneous communicative framework for HCI where designers,
users, and computer artifacts are brought together can be used to inspect the nature
and the quality of design processes and products. The latter can be further analyzed
both per se and in situthat is, outside and inside the contexts of use. Since com-
munication is such a pervasive activity in this semiotic perspective, we have the tools
to answer a number of questions. For instance, we can analyze the following:

What signs are being used to compose the signication system that will carry the
one-shot message to users, and also support ongoing communication between the
user and the designers deputy?
30 Chapter 1

How does the signication system encode design intent? Is this encoding likely to
be understood by the users? Why?

Once the users understand design intent, how can they put this intent into oper-
ation for their own benet?

What kinds of semiosic processes triggered by the signs that compose the
engineered signication system does the designer anticipate? Which ones can the
designers deputy process? Which ones can it not?

If the users semiosic processes evolve in directions that have not been anticipated
by the designer, which signs will express this communicative breakdown? What
remedial resources can be used to reestablish productive communication? Can users
extend the signication system? How?
Finally, by including designers in the unit of investigation of HCI-related phe-
nomena, semiotic engineering opens the door to an examination of its own limita-
tions. In the process of analyzing this specic kind of computer-mediated human
communication, designers may realize that they need tools, knowledge, or other
resources that this theory is not able to provide. And as soon as they realize it, they
can look for other theories that will complement semiotic engineering (i.e., provide
the additions that will lead to a complete fulllment of needs), supplement it (i.e.,
provide additions that will lead to an increased fulllment of needs), or simply
replace it (because others can yield better results altogether).
1.6 Expected Contributions for Professional Practice
Novel theoretical approaches to HCI have been repeatedly called for to advance
research and contribute to improve the quality of information technology artifacts
(see, e.g., opinions expressed in Barnard et al. 2000; Hollan, Hutchins, and
Kirsh 2000; Shneiderman et al. 2002; Sutcliffe 2000). As we have already men-
tioned, however, the latter has been the object of some debate, since studies have
reported that practitioners disagree with the view that theoretical research can have
positive and timely impacts on the quality of HCI product design (e.g., Gugerty
[1993]).
Donald Schn (1983) reviewed and critiqued alternative views on the relation-
ship between scientic and practical knowledge, and proposed that there are two
paradigmatic positions in this eld. One is the technical rationality perspective,
according to which practical knowledge is applied science. In this view, technical
Introduction 31
professionals should be trained in basic sciences that provide them with generally
applicable problem-solving methods, and then develop the necessary skills to select
and apply the best-t method to the problems they are likely to encounter (Simon
1981). The other is a reection-in-action perspective, according to which practical
knowledge involves the capacity to name the elements present in problematic situ-
ations and frame them as problems. These problems are often unique and usually
unstable, unlike the general kinds of problems assumed to exist in the other para-
digm. In the reection-in-action view, technical professionals must be equipped with
epistemological tools that can help them raise useful hypotheses, experiment with
different candidate solutions and evaluate results. As a consequence, Schn suggests
that the education of professional designers can benet from more epistemic
approaches to practice. Schns message to HCI designers in particular (Schn and
Bennett 1996), is that they should ask themselves what is the artifact they are about
to design, and not only how they can make the artifact usable. The answer to the
rst question, as in the case of engineering models, for example, can be found by
interacting with [a] model, getting surprising results, trying to make sense of the
results, and then inventing new strategies of action on the basis of the new inter-
pretation (181).
If we look at HCI research over the last decade or so, we see that a very large
portion of the results it has produced falls in the category of predictive methods or
explanatory models of users behavior (Shneiderman et al. 2002). The products of
research have typically taken the form of design guidelines and heuristic rules.
Tacitly assuming that designers are faced with recurring problem types, many
researchers struggle to articulate guidelines and rules into generalizable principles.
Principles, in their turn, can be used to support the methods with which design prob-
lems can be skillfully resolved. Perhaps the most enduring instance of how theories
can be used in this direction is Card, Moran, and Newells model (1983) of the
human information processor. For nearly two decades, this theory has inuenced a
considerable portion of HCI designers who view interaction as a cognitive problem
whose solution can be found with the application of appropriate task and user
modeling techniques derived from the original theory or some of its derivatives.
There are, however, some problems with the technical rationality perspective.
Experimental validation of the methods derived from the various theoretical stances
is difcult if not impossible to carry out. Keeping rigorous control over statistically
relevant corpora of design situations in current HCI professional practice would not
only involve enormous costs (having different design teams developing the same
32 Chapter 1
real world application by using different comparable methods; tracing every step
of their progress; and performing statistically signicant product evaluation tests
in the end), but involve methodologically dubious procedures (deciding whether
an implemented program strictly follows a design specication; deciding whether
a particular design specication represents all dimensions of design intent and
content).
Another kind of difculty with technical rationality is that the purposes and
requirements that justify and trigger the design of various interactive software arti-
facts are constantly changing. Many HCI practitioners believe that users dont know
what they want. But this isnt necessarily the case. Users may know very well what
they want and yet change their minds every time they are asked about requirements
and purposes. As everybody else, the more they think about the problem, the more
they understand about it. So, new insights are constantly arising from their reec-
tion. This situation makes it again very difcult to practice HCI design as applied
science, because the problem to apply it to is intrinsically unstable (and not
amenable to the kinds of generalizations targeted by technical rationality).
In spite of these difculties, and the resonance of Schns ideas in a number of
HCI research centers, researchers have not turned to epistemic theories of HCI, or
to producing epistemic tools. An epistemic tool is one that is not used to yield
directly the answer to the problem, but to increase the problem-solvers under-
standing of the problem itself and the implications it brings about. In the same spirit
as Kirsh and Maglios account of epistemic actions performed by Tetris players
(1995), epistemic design tools are those that will not necessarily address the problem
solution, but the problems space and nature, and the constraints on candidate solu-
tions for it. Suchmans ideas (1987) and Winograd and Floress account (1986) both
called attention to the fact that intelligent human behavior uses freely exploratory
interpretation of signs belonging to the context of activity in order to take
goal-related action. And good computer tools should help people do this more and
better.
Given the objectives that we set out to achieve with semiotic engineering, we
can say that the main theoretic contribution we intend to make is to address the
epistemic needs of both designers and users when communicating through or with
computer-based technologies. With a semiotic perspective on HCI, we intend to
open the road for eventually producing epistemic design tools that will help design-
ers deal with knowledge itself. The reective nature of semiotic engineering is
particularly suited for the task. It can be easily articulated with Schns
Introduction 33
reection-in-action perspective on design, and can strive to compensate the dif-
culties brought about by the technical rationality perspective by raising the design-
ers epistemological awareness on what they are doing and on how this affects what
users are doing.
This book is organized in three parts. Part I offers an introductory overview of
our goals and assumptions, followed by a concise presentation of some theoretical
concepts used in semiotics that are necessary for understanding semiotic engineer-
ing. It ends with a detailed presentation of our theory of HCI. Part II shows how
the theory can be applied to inform the evaluation and design of HCI products in
three specic contexts. I start with communicability evaluation and the construc-
tion of online help systems. Then I explore customization and extension of appli-
cations in the context of end-user programming. And I conclude the second part of
the book with a discussion of the semiotic engineering of multi-user applications.
Part III contains a single chapter, in which I share with readers my questions, beliefs
and expectations about the potential and the immediate opportunities for research
in semiotic engineering. Because the theory is itself a sign that readers may add to
their ongoing semiosis, there is no ultimate conclusion or prediction about what
semiotic engineering means. This book is only my current discourse about what it
means to me, hoping that it will be a useful and stimulating reading for those who
like to think about HCI.
34 Chapter 1
2
Fundamental Concepts in Semiotics
In this chapter I present a small set of semiotic concepts, all related to each other,
which form the theoretical basis of semiotic engineering. The chapter is neither a
summary of any existing semiotic theory, nor is it a comprehensive overview of
semiotics as an entire eld of study or discipline. It is rather a reading of certain
semiotic theories, aimed at building the foundation for a semiotic theory of HCI.
All the selected concepts play a fundamental role in semiotic engineering. Other
unmentioned concepts have been or may be useful for advancing theoretical aspects
of HCI, but the selected set is intentionally small and cohesive (ideally minimal), so
that the targeted theory can be more easily understood.
Semiotics covers a vast territory of scholarly theorization and debate. It is
generally concerned with meaning-making and representation in many forms
(Chandler 2002) and with everything that can be taken as a sign (Eco 1976).
Depending on the domain of meanings, on meaning-makers and meaning-takers,
semiotics crosses disciplinary intersections such as those with psychology, anthro-
pology, biology, logic, linguistics, philosophy, and many other applied areas of study
and professional practice, like media studies and even software engineering. The
themes of semiotic interest have been the object of investigation for as long as phi-
losophy exists. A search for the meaning of meaning and the nature and purpose
of representations is the legacy of Greek philosophers to all those engaged or inter-
ested in asking what they know, and why or how it is that they know what they
know. Therefore semiotic engineering favors a philosophical perspective on investi-
gating, teaching, designing, evaluating, and even practicing HCI.
I introduce and illustrate the following relevant topics:

semiology and semiotics

signs, semiosis, and abduction

sign classications, their origin, and their use

pragmatics, speech acts, and culture

sign production, communication, and discursive competence

metaphors and metonymies

All denitions will be given in a format that suits the application of the corre-
sponding concepts in HCI. Consequently, they do not reect existing (and often
heated) contention among theorists who have opposing views or see further dis-
tinctions in their account of sign-related phenomena. My goal is to be consistent
and concise in presenting a semiotic interpretation of HCI, not to provide an exhaus-
tive account of all possible interpretations.
It has already been said that semioticians obscure rather than illuminate a novices
understanding of the eld (Chandler 2002). One of the characteristics in their
writing that thoroughly contributed to this reputation is their esoteric use of ter-
minology. In this introduction to semiotics, I have made a deliberate effort to clear
at least some terminological obstacles. Whenever possible, more familiar words have
been used to convey the essence of the theory.
2.1 Semiology and Semiotics
Modern semiotics is the result of the independent intellectual efforts of two remark-
able thinkers, Ferdinand de Saussure (18571913) and Charles Sanders Peirce
(18391914). Although they were contemporaries and both devoted to unveiling
the mysteries of signicationone in Europe, the other in North Americathey
developed distinct and independent theories. De Saussures semiology appeared in
Switzerland, out of his interest in the study of language and linguistics (de Saussure
1972). Peirces semiotic, known today as semiotics, appeared in the United States,
out of his interest in the study of logic and the philosophy of science. Much of the
variety in semiotics today is due to the individual power of de Saussures and Peirces
thinking and the breadth of their theories along such different lines.
De Saussures theory of signs emerged from his perspective on language as a
system of arbitrary signifying units. His main concern was to account for the general
mechanisms by which such unitsor signswere organized into abstract constructs
that individuals put to use in a variety of social contexts. He called the abstract
system langue, and its realization in specic instances of sign usage, parole. This
distinction opened the way to the structural study of language as an autonomous
36 Chapter 2
object of investigation, regardless of the particular physical or psychosocial con-
straints to which individual behavior is exposed (like mental capacity, speech pro-
duction limitations, or cultural regulations). Langue was viewed as the result of
consistent distinctions and principled distribution of system units whose value
(atomic or combinatorial) was determined by the values of all other units in
the system. As a consequence, the study of language as langue required, from a
Saussurean perspective, the concurrent examination of all units (simple and com-
pound) in the system. This holistic synchronic view of language provided the
appropriate background for the study of linguistic varieties (individual, social, geo-
graphic, historical, rhetorical, etc.) and served as inspiration for structuralisma
particular perspective on scientic analysis made famous by the anthropological
studies of Claude Lvi-Strauss and the psychoanalytic theories of Jacques Lacan, for
example.
Another capital feature of the Saussurean signs is that they were postulated as
arbitrarythat is, not causally determined by that which they signify. This postu-
lation underlines the nature of representation in its widest sense. To re-present
something is not to replicate or re-produce it, keeping all the determinations
applicable in its contexts of occurrence in physical or mental reality. Rather, to re-
present something is to bring it up again in another dimension (the representa-
tional one), where causation and truth yield to the possibilities of signication. One
of the extreme consequences of the arbitrariness of signs with respect to what they
signify is the possibility of using language to tell lies. Lies are only possible because
representations are not linguistic reinstantiations of the world. This apparently
obvious point about the noncausal nature of signs reemphasized the independence
between meaning and truth, a recurrent philosophical issue throughout the centuries
that is often overlooked both in theory and practice.
One of the various connections that can be established between the basics of semi-
ology and HCI is that, although de Saussures focal system of signication was natural
language, he explicitly stated that his theoretical framework could be applied to the
analysis of any other system. Thus, computer applications interfaces can be viewed
as an object of semiological investigation. However, some of the structuralist concepts
presented previously are substantially modied in a computer environment. For
example, the distinction between langue and parole virtually doesnt exist from the
perspective of a computer program. Unlike humans, a computer program does not
produce nongrammatical sentences or mispronounced words because it is under psy-
chological pressure or physical stress. Computer language generation is not parole,
Fundamental Concepts in Semiotics 37
but an algorithmic instantiation of the rules of langue. Attempts to incorporate indi-
vidual variations of parole into the computers langue results in major complexities
for design, in particular because parole is not just a peculiar individual way to use a
socially established system of signs. It is the very seat of psychosocial and cultural
determinations of sign systems, whose algebraic abstraction, so to speak, is the langue.
Therefore, as soon as psychosocial and cultural determinations are merged with alge-
braic specications of computer languages, the theoretical distinction is lost, and with
it the ability to account for (and simulate) our fault tolerance, and awareness of it, in
using language and other natural sign systems.
Another illuminating connection between HCI and de Saussures semiology is
related to the arbitrariness of signs. As has been mentioned, natural language and
other human sign systems allow you to represent things that are not true, such as
(intentional or unintentional) lies and wishes. It is thus related to your capacity to
imagine and create things that you do not verify in reality. However, computer
systems cannot produce signs unless they are the result of some algorithmic sym-
bolic manipulation, and are thus causally related to base signs programmed into
applications. Therefore, not only are interactive systems unable to wonder, imagine,
or lie in any legitimate sense of these words, but also they cannot appropriately
handle a users imaginative or humorous behavior. This is an important realization
for the design of intelligent and adaptive user interfaces. Intelligent computer pro-
grams cannot be expected to handle the full range of human signication processes,
and users should be aware of where limitations lie.
The Peircean perspective on signs was not motivated by natural language, but by
Peirces interest in logic and epistemology. Peirces philosophical perspective was
much broader than de Saussures, so much so that among the themes of his writ-
ings are the nature of mind and knowledge, the appropriate methods for scientic
inquiry, the basic phenomenological categories that mind can perceive, and even the
inuence of love or sympathy on the evolution of the universe (see The Peirce Edition
Project). Semiotics was then geared toward epistemological, rather than linguistic,
concerns. Peirce was convinced that all advanced forms of thought (especially sci-
entic thought) were fundamentally based on conventions and rule-based represen-
tations. So, to him, an epistemology of science was ultimately an account of such
conventions and rules, an account of the underlying signication systems in which
and through which knowledge was built.
His denition of a signanything that stands for something else, to somebody,
in some respect or capacity (Peirce, 19311958)underlined the fundamental role
38 Chapter 2
of interpretation (what somebody takes something to mean) in semiotics. Nothing
is a sign unless (or until) it is interpreted by somebody. It follows from this deni-
tion that the same signs may have very different valid meanings, that mutual intel-
ligibility widely depends on cultural conventions and mechanisms to negotiate
shared meanings, and that ultimately there is no such thing as the meaning of a sign
(not even in the equivalent of de Saussures langue). The consequences of this theory,
not only for logic and linguistics, but also for all disciplines where meaning is
central, cannot be underestimated. For example, in a semiotic perspective meaning
cannot be framed as a xed and permanent entityit is a process. Consequently,
all theories built on the existence of unchanging meaningful objects suddenly
become incompatible with this perspective and are automatically challenged by it.
However, the concept of a continuing semiotic interpretation of signs does not stand
unless it can be made compatible with our nite minds and nite resources. There-
fore, an account of how and why the ongoing interpretive process is halted and
(temporarily) instantiated as the meaning of a sign was a crucial requirement for
the scientic plausibility of Peirces theory.
Peirces response to this challenge came as a method for hypothetical reasoning,
also known as abductive reasoning, or simply abduction. The gist of this method
(which will be presented in further detail in this chapter) was the idea that mind
assigns meaning to things by building plausible hypotheses about the sign that is
taken to represent them. As long as these hypotheses are conrmed by positive evi-
dence, they concur to build the meaning of the sign being interpreted. If and when
negative evidence is found, hypotheses (and meanings) are modied or replaced by
more plausible ones. For instance, when going back home at the end of the day, if
you see that the lights of your house are on you take this to mean (i.e., you assume,
or hypothesize) that there is somebody home. But if you open the door and see
nobody, then you may assume (or take the lights on to mean) that somebody has
forgotten to turn the lights off before leaving. In the absence of counterevidence,
you stabilize your interpretive process at this point. It may be resumed, however, if
a few hours later everybody else in the family is absolutely sure that they turned
the lights off before they left, or if you realize that some of your valuables are missing
(in which case the lights on would be a sign of burglary).
With more or less explicit acknowledgement from researchers, Peirces theories
have contributed to articial intelligence and machine learning, especially with
respect to frame-like knowledge representation techniques (Eco 1984; Eco,
Santambrogio, and Violi 1988) and automatic reasoning methods (Josephson and
Fundamental Concepts in Semiotics 39
Josephson 1994). Their fundamental contribution to HCI was pointed out by Mihai
Nadin as early as 1988 (Nadin 1988), but the only relatively popular (and often
misunderstood) use of Peircean semiotics concepts in HCI is his classication of
signs into icons, indices, and symbols. As will be seen in other parts of this chapter
and other chapters of this book, semiotic engineering makes extensive use of
Peircean concepts in its account of HCI design and evaluation. This is why I priv-
ilege the term semiotics over semiology.
Other semioticians have inuenced our conception of semiotic engineering as a
theory of HCI. First comes Umberto Ecodescribed by Chandler (2002) as a the-
orist who bridges both the structuralist and the Peircean traditions. His view of
semiotics as the logic of culture (Eco 1976, 3), as its name suggests, brings
together the human (culture) and the computational (logic) underpinnings of HCI
and supports a new breed of theoretical formulations and methodological tools for
researchers and practitioners. Additionally, Charles Morriss view (1938, 2) of semi-
otics as an instrument for science has helped me elaborate relevant epistemolog-
ical aspects of semiotic engineering. Although Morriss perspective does not fully
acknowledge its Peircean origins, his reading and presentation of Peirces theory
is clearer and more direct than its sources. Thus, Morris has provided insightful
interpretations of how semiotics can be used as a metalanguage for scientic
descriptions.
2.2 Signs, Semiosis, and Abduction
Signs, semiosis, and abduction are the three most basic semiotic concepts used in
semiotic engineering. Signs, as we know, have been dened by Peirce as anything
that stands for something else, to somebody, in some respect or capacity (Peirce,
19311958). However, unlike many of his predecessors throughout the history of
philosophy, Peirce did not describe the structure of a sign as twofold. Unlike de
Saussure, for example, who followed this tradition and proposed that signs were a
relational structure binding a representation (a perceptible form) to what it repre-
sents (a conceivable content), Peirce proposed that signs had a threefold structure
and, as seen previously, functioned as a process. The three constituents of the
Peircean sign were the representation (called the representamen), its referent (called
the object), and its meaning (called the interpretant). The triangular representation
of this structure has become popular in semiotics. In gure 2.1 two important
aspects of this structure are underlined. First, that there is no necessary direct con-
40 Chapter 2
nection between a representation and that which it refers to. This connection may
be there (as is the case of a photograph on a drivers license), but it doesnt have
to. Second, meaning is always a mediator between a representation and what it
refers to. In other words, there is actually no representation unless there is a meaning
binding it to what it refers to, which is another way of saying that there is no sign
unless some interpreter takes a representation to mean something else. So a sign
requires the concurrent presence of these three constituents. From a semiotic per-
spective one cannot talk about representations without talking about their referent
and meaning. Nothing is a representation except by virtue of a sign that ties it to
referent and meaning.
We intuitively know that representations have meaning and that it doesnt make
much sense to talk about them independent of what they mean. But the news
brought by Peirce was that representations and meanings are still not enough (where
he diverges from de Saussures opinion). The presence of three constituents directs
our attention to much confusion about what representations really mean. A com-
plete understanding of this nuance cannot be reached without an appropriate de-
nition of the interpretant and its connection to semiosis.
The meaning of a representation, technically called the interpretant, or by exten-
sion the meaning of the sign, is dened by Peirce as another sign. Consequently, for
every sign there is another sign that corresponds to its meaning. Interpretation being
the process by which meanings are associated to signs, it follows from Peirces
Fundamental Concepts in Semiotics 41
Figure 2.1
The structure of the Peircean sign.
denition that this process is theoretically innite. Although the idea is intuitively
contradictory with the niteness of human minds, we can have a glance at its psy-
chological plausibility in reective mental processes. For instance, the more we think
about some signicant thing, the more meanings we associate to it (essential or acci-
dental). Therefore, there are indenitely many meanings (a psychologically more
palatable version of innity) to a sign, and it is impossible to predict the exact path
and length of meaningful associations that will be made during interpretation. This
interpretive process has been originally called semiosis, and later unlimited semio-
sis (Eco 1984).
In order to illustrate the practical implications of unlimited semiosis, let us assume
that a customer goes into a small shop and sees the sign of a hammer ( ) hanging
from one of the shelves where goods are displayed. We may rightfully assume that
the representation refers to a hammer, but we shouldnt assume that the meaning
of is a hammer. From a sign-makers perspective (e.g., the owner of the shop),
the meaning of this representation may well include such signs as we sell
hammers, here is where hammers can be found, have a look at our hammers,
and so on. From a sign-takers perspective (e.g., the customer who has just come
into the shop), the may mean not only such things as they sell hammers or
this is where their hammers are, but also things like they sell nails or I need
to buy a hammer. Notice that the latter two are associated with as a result of
inference and episodic memory associations, not of strictly semantic relations (a
popular source of denitions for the meaning of representations in other theories).
The particular kind of inference involved in interpreting as a sign that the
small shop also sells nails is a case of abductive reasoning. Compared to deductive
reasoning (see gure 2.2), abduction is practically a reversed reasoning path. Deduc-
tion cannot start unless there is a set of known rules and facts about the world. Not
only are they known, but they are also true and unchanging. For instance, it is a
known fact that all physical objects fall (a nave version of the law of gravity).
Because a hammer is a physical object, if we let one go from our hand it will surely
fall. Abduction, however, does not require that rules and facts be known and true
from the beginning. Rules and facts are hypothesized whenever they can contribute
to explaining (or making sense of) some observation in the world. In the small shop
example, the presence of can be explained, or makes sense, if we can assume the
existence of a rule that says that all shops displaying the sign of a hammer sell
hammers. If this rule existed, given that this particular one displays such a sign then
it makes sense to think that means that the shop does sell hammers. Notice that
the hypothesized rule is not comparable to the law of gravity. It is only a plausible
42 Chapter 2
(commonsense) assumption. The sign might in fact mean something elsethat
the shop repairs broken things, that the goods on that shelf are unbreakable, or
whatever the sign-makers might have intended. Of course, the greater the distance
between the intended meaning and the meaning that results from the most plausi-
ble rules, the lesser the chances that the meaning-takers will understand the sign as
intended.
Assuming that means that the shop also sells nails is a more elaborate abduc-
tion. It is the result of more than one abductive cycle. The rst one leads to the con-
clusion that the shop sells hammers. The second one assumes that all shops that sell
hammers also sell nails. Because this one (presumably) sells hammers, it must then
sell nails as well. Once again, there is not a known true rule in the world saying so,
but the meaning-takers previous experience with signs and their meaning (semio-
sis) allows him or her to raise these hypotheses. The interesting fact, for computer
science, is that abductive reasoning does not depend on the existence of known and
stable primitives or axioms. The elements that function as primitives and axioms
are hypothesized, produced on demand, and tested for suitability against the evi-
dence collected by the human interpreter (or reasoner). This mode of operation
Fundamental Concepts in Semiotics 43
Figure 2.2
Abductive reasoning compared to deductive reasoning.
cannot be accommodated by the classical Turing machine model, where symbol pro-
cessing fundamentally depends on primitive symbols from which others can be
causally derived by the effect of generative rules (for interesting critiques of the
status of symbols and meaning and the limitations of the Turing machine model for
computation, see Fetzer 1988 and Wegner 1995). So HCI is challenged by irre-
ducibly different modes of sign interpretation and sign production. The conse-
quences of such disparity for the communicative processes taking place in HCI
constitute a central theme of semiotic engineering.
HCI provides us with clear evidence of users engaged in abductive reasoning. The
following is an anecdotal scenario reported to the author by a user of Microsoft
Excel 2002. In it we can observe the power of sign interpretation and the kinds of plau-
sible assumptions that users may make while interacting with computer applications.
44 Chapter 2
John had prepared a spreadsheet with a large volume of data about his divisions
performance for the last three months. He then wanted to print it and take it to
his manager in order to discuss some of his ndings. He had to squeeze some of
the columns in order to make some critical portions of the spreadsheet t onto a
single printed page. However, after having made a couple of changes to the
original format, he suddenly realized that various ### signs had appeared in the
printable format (see gure 2.3 for a partial snapshot of Johns screen). Being
relatively familiar with various Excel error codes, he took the ### to be error
messages. But he was puzzled because he was sure that the spreadsheet had no
errors before he started formatting it. Could the formatting have introduced errors
in the spreadsheet?
In order to remedy the situation he restored the format he was using before he
decided to print the spreadsheet (which took him a while), but to his surprise
he could not see the ### signs anymore. And he could not quite remember where
they had appearedthere were many of them. He concluded that he had
inadvertently ruined something during the formatting process. So, he started it
all over again, only to nd the ### signs once more. Discouraged by the
experience, he concluded that some formatting operations affect the data in large
spreadsheets.
Johns conclusion has its logic. Excel indeed uses the # sign to signal error messages
such as, for example, a division by zero#DIV/0!. And his previous experience with
the # sign was all about errors in numerical operations. So he did not hypothesize that
Fundamental Concepts in Semiotics 45
Figure 2.3
The # sign in Microsoft Excel 2002 visualization. Screen shot reprinted by permission from
Microsoft Corporation.
another kind of errora data visualization errorwas the case. Because John
squeezed columns to t in one page, the space necessary to visualize some of the data
became smaller. Consequently, the ### sign appeared. But since John hypothesized
the wrong rule (i.e., that data were lost), he never tried to solve the problem by using
formatting tools (e.g., by using another font type, such as Arial Narrow, in his par-
ticular case). This would have enabled perfect visualization in the small scale he
wanted to use. His semiosis took him away from the available solution. And Excels
help system was of no use in this case. The Whats this? tool, graphically represented
as , when placed on the ### sign, simply said No help topic is associated with this
item. It failed to interpret this as a reference to ###. So although the tools to solve
the problem were there and were not at all difcult to use (John himself says he uses
the font type tool regularly), the right kinds of tools needed to support the formula-
tion of appropriate hypotheses in this case were not.
2.3 Sign Classications, Their Origin, and Their Use
One of the most popular references to Peircean semiotics in HCI is, as mentioned
earlier, his famous classication of signs into icons, indices, and symbols. The liberal
use of the word icon in HCI jargon, however, has contributed extensively to creat-
ing a great confusion about the original denition and purpose of this classication.
A semiotic theory of HCI must only talk about sign classications if they can be
shown to enhance its descriptive, predictive, or explanatory power. In order to
decide whether such classications are useful or not, we should examine their
origins.
Peirces semiotics was intrinsically related to his greater philosophical project,
which included fundamental postulates about the basic categories according to
which all possible knowledge can be classied. Aristotle (1980), for instance, pos-
tulated the existence of ten categories: substance, quantity, quality, relation, place,
time, position, possession, activity, and passivity. Radically committed to his epis-
temology (the study of what there is to be known and how such knowledge origi-
nates, expands, and collapses), Peirce postulated the existence of only three
categories of semiotic interestrstness, secondness, and thirdness. The minimalist
names he assigned to them serve to dismiss confusions with other sets of categories
known to philosophers.
Firstness is the category of undifferentiated qualitative experience. It encompasses
all the phenomena that we are aware of experiencing but do not discriminate from,
or associate with, anything else. We can intuitively align this category to sensations,
46 Chapter 2
perceptions, and gut feelings that we cannot put into words or express rationally. Sec-
ondness is the category of strict associations between two phenomena. Our capacity
to relate one thing to another requires that we perceive an invariant or commonality
between them. The name or reason of the invariant does not have to be known (for it
would constitute a third thing in the observed relation); only the fact that there is
an association is known. Thirdness is the category of mediated relations, involving at
least three concepts. Going back to secondness, if the invariant between any two
related things is itself a thing to the knower, than we have a case of thirdness. This
is the category of rationality par excellence, because it allows us to formulate princi-
ples for relating things, for naming classes of things, and so on.
Concrete examples of rstness are often provided by young children when they
dont feel well. Parents often have trouble trying to understand what is wrong with
them in such situations, because the children do not know how to verbalize it. Fol-
lowing this line of illustration, the kids give signs that something is wrongthey
cry, they are irritable, they have temperature, for instance. All of these are instances
of secondness for parents (and for some kids, too). They are symptoms that some-
thing is wrong. And it is not until parents assign some acceptable meaning to these
symptoms that we have a case of thirdness. The assigned meaning may be wrong,
as we know, for it is typically the result of abductive (hypothetical) reasoningnot
of a deduction. So, thirdness is the legitimate seat of rationality because it takes at
least three phenomena to complete an inferential path (as seen in gure 2.2).
Peirces denition of icons, indices, and symbols rests upon these three phenom-
enological categories and is much more elaborate than the denitions usually
adopted in HCI. Originally, the threefold classication refers to how representations
evoke their referents (not their meaning)that is, to how the representamen man-
ifests the object of a sign. Iconic signs are those where the representation brings out
the rstness of its referent. Looser interpretations of this denition have popular-
ized the view (used by Peirce himself as illustration) that icons are signs that resem-
ble or look like that to which they refer. A preference for visual resemblance
over all other possible sorts resemblance has contributed to the current equivocal
view that an icon is a visual representation of something. Technically, an icon is a
kind of representation that brings up the undifferentiated qualitative experience of
sensing the referent. A sound may be an iconic representation (and we technically
dont need the word earcon to refer to it), and so may a gesture or mimic.
Indexical signs are those where the representation brings out the secondness of
its referent. The proverbial examples of smoke representing re, snow representing
cold, and so on, can easily illustrate the idea of secondness for causal associations
Fundamental Concepts in Semiotics 47
between two concepts. But secondness does not refer only to causal associations.
The term contiguity has been used to account for cases of co-occurrence in general.
Thus, the representation of a train can bring out secondness if it refers to a railway,
and that of a printer can bring out secondness if it refers to printing documents.
Symbolic signs are those in which the representation brings out the thirdness of
its referent. Most of the words in any natural language are instances of symbolic
representation. They are the most genuine case of de Saussures arbitrariness of
signs. They result from a regulating convention that relates them to their referent
by virtue of an established meaning (see gure 2.1). There are other symbolic rep-
resentations that are not words. For example, refers to innity by virtue of a
conventionalized notion of what innity means. Some other symbolic represen-
tations, like , are actually quite visual (which does not turn them into an icon).
The sign refers to a data le that some particular application can process. The
variations of the verbal correlates of this visual representation are a telltale sign of
its constitutive thirdness. Document, le, and even project are words fre-
quently combined with New in menu options whose meaning systematically cor-
responds to that of this visual representation. is thus not technically an icon
because the rstness it brings about is only that of a blank page, something only
metaphorically associated to what users know it to mean.
Peirces classication can thus, in its original sense, help us understand the semi-
otic complexity involved in producing or interpreting signs. It can be used to achieve
the same kinds of analysis that Hutchins, Hollan, and Norman (1986) have
achieved, for example, with their notion of articulatory and semantic directness.
However, because the meaning (interpretant) of a sign is itself another sign, Peirces
categories can be applied recursively to examine the meanders of semiosic paths in
a deeper degree of nesse than is achieved in traditional cognitive approaches such
as that of Hutchins and colleagues. Semantic directness is a function of the distance
between a users conception of a goal or subgoal he wants to achieve and the cor-
responding structure of functions and procedures in the abstract model of the appli-
cation. For instance, in most Windows operating systems, the semantic directness
involved in creating folders is greater than the semantic directness involved in par-
titioning a hard disk. In order to understand why partitioning a disk is much more
complicated than creating folders, and why we usually partition a disk before we
start installing applications and storing data, we must abandon the desktop
metaphor. In a real desktop or ofce environment, we can divide and organize space
quite freely by creating subdivisions on empty and occupied areas. In the computer
48 Chapter 2
version of a desktop, this can be done with folders and subfolders (in accordance
with the metaphor), but usually not with hard disks (containers of all folders). The
greater nesse of a semiotic analysis stems from the meaning of a sign being a func-
tion of ongoing semiosis. In other words, every sign appearing in the same domain,
context, or mind as the one being interpreted affects the course of its interpretation.
Thus, whereas from a semiotic perspective all the representations and meanings
appearing in an application are necessarily involved in the analysis, the scope of
analysis considered in semantic directness (as dened by Hutchins, Hollan, and
Norman [1986]) is limited to that of the users current goal or subgoal. Moreover,
a semiotic analysis does not distinguish between semantic and articulatory direct-
ness. Articulatory directness is a function of the distance between the structure of
functions and procedures in the abstract model of the application and the interface
actions (the actual production of sign combinations) necessary to execute the users
goals or subgoals. The structure of the Peircean sign (which binds representations,
referents, meanings, and meaning associations together) creates a continuum where
semantics and sign articulation cannot be separated.
Peirces sign classications based on his fundamental categories can highlight the
aspects Hutchins, Hollan, and Norman (1986) were interested in. The notion of
distance that serves to establish semantic and articulatory directness can be replaced
by rstness, secondness, and thirdness. Let us assume that a user of Treemap has a
data set with the United States population for several years between 1981 and 1997.
Her goal is to nd out which state, in that period of time, had the largest popula-
tion growth in absolute values. Treemap allows her to use different visualization
styles and manipulations to interpret and compare data. An analysis of the seman-
tic directness for achieving her goal with Treemap will show that there is no direct
translation of her intent into a function that will return the result of a direct query
(e.g., for all Y = state and all X = pop(Y in 1997) - pop(Y in 1981), show(sort(list
of all X))). The user must translate her goal into the signs appearing in the seman-
tic model of Treemap (e.g., map color to attribute, map size to attribute, lter
range of values, visualize ltered values, and so on). What she decides to do
(according to her personal interpretation of the application and of how it can help
her achieve her goal) is to use lters. She guesses that the most signicant changes
would happen among the most populated states (an abductive step). So she starts
by visualizing the ve most populated states for every year for which data is
available. In 1981, the ve most populated states (in screen-reading order) are
California, Texas, Illinois, New York, and Pennsylvania (see gure 2.4). In 1992,
Fundamental Concepts in Semiotics 49
however, the ve most populated states are California, Texas, Florida, New York,
and Pennsylvania (see gure 2.5). She repeats the ltering for all years in the data
set and nds out that Florida remains among the most populated states in all other
visualizations.
The articulatory directness in Treemap (as the very idea of direct manipulation
predicts [Shneiderman 1983]) is very high. Filtering data involves only a manipula-
tion of sliders in the control panel on the right-hand side of the screen. But, unlike
what a semiotic analysis can do, semantic and articulatory directness alone cannot
account for what the user does next. She starts exploring direct manipulation to
expand the scope of ltering and to discover, for example, the order in which states
50 Chapter 2
Figure 2.4
Visualizing the ve most populated states in 1981 with Treemap 3.2. Screen shot reprinted
by courtesy of HCIL, University of Maryland, College Park (http://www.cs.umd.edu/hcil/
Treemap/).
are visualized for each year (less populated states are darkened after the most
populated ones). Her manipulation of available signs in the interface shows her
that in 1981, for example, Florida was darkened even before Ohio. But in 1992,
Ohio was darkened before Florida (which she takes as a sign of Floridas popula-
tion growth, although this conclusion is not warranted by deductive reasoning
methods, since Ohios population could have technically decreased). She also nds
out that, according to her data set, in 1981, when Florida was the seventh most
populated state in the United States, Pennsylvania was the fourth. However, in 1992,
Florida was the fourth most populated state in the United States, ahead of
Pennsylvania.
Fundamental Concepts in Semiotics 51
Figure 2.5
Visualizing the ve most populated states in 1992 with Treemap 3.2. Screen shot reprinted
by courtesy of HCIL, University of Maryland, College Park (http://www.cs.umd.edu/hcil/
Treemap/).
When analyzed from a cognitive perspective using semantic and articulatory
directness, Treemap seems to be very easy to use once the users succeed in trans-
lating their goals into the particular strategies that are implemented in this appli-
cation, thanks to the specic visualization purposes that it aims to facilitate and
demonstrate. A semiotic analysis can nevertheless go some steps further and account
for a wider scope of interaction. For instance, why was it that the user chose to
start from screening the ve most populated states? Abductive reasoning and the
process of generating plausible hypotheses can explain her use of common-sense
information. Since her goal is to analyze growth in absolute terms, her rst guess
is that the answer will be found among the most populated states. This path leads
her to spot the growth of Floridas population for that period of time. And once
her semiosic path includes screening and ltering to produce other signs, she
explores the idea of ranking, which eventually conrms (to her) that Florida is not
only the state she is looking for, but that it had remarkable growth rates between
1981 and 1997. As semiosis goes on, the user is likely to ask herself why Florida
became so populated, or maybe nd out when Texass population became larger
than New Yorks, and so on.
Further semiotic analysis reveals that direct manipulation explores secondness
(the contiguity aspects of representation with respect to its referent). It explains why
semantic directness is lower in Treemap: The thirdness involved in the notion of
most populated state is not privileged by the representation style of this applica-
tions interface. Moreover, a semiotic analysis can directly help us explore how
technology inuences problem-solving strategies and knowledge discovery. Just for
the sake of exemplication, we can say that interfaces that choose to emphasize
secondness over thirdness are likely to promote associative knowledge discovery
and manipulation with greater ease than conceptual knowledge discovery and
manipulation.
The idea that category-based semiotic analysis can explain (and anticipate) the
success of some HCI styles and strategies over others has been explored in a case
study about learningware (de Souza and Sedig 2001). Super Tangrams is an appli-
cation built to help children learn and retain the knowledge involved in translation,
rotation, and mirroring of geometric objects (Sedig, Klawe, and Westrom 2001).
Three different styles of interactiondirect object manipulation (DOM), direct
concept manipulation (DCM), and reective direct concept manipulation
(RDCM)promote different conceptualizations (semiosis) about 2-D operations on
52 Chapter 2
geometric objects and can make it easier or harder for students to learn the princi-
ples of Euclidean geometry.
Super Tangrams DOM interface works like most graphic editors (see gure 2.6a).
Children can rotate objects to the left or right. They can ip them vertically or hor-
izontally. And they can drag and drop them anywhere on the canvas. In accordance
with ndings from previous studies, Sedig and colleagues found out that children
using the DOM interface were often confused about geometric operations. For
instance, the drag-and-drop interaction often concealed the straight path constraint
for geometric translations, leading children to think that geometric translation can
follow an arbitrary curvilinear path.
DCM calls the childrens attention to geometric concepts (vertices, axes, angles)
by representing them visually and by allowing children to manipulate them instead
of the generic shapes appearing in DOM (see gure 2.6b). This style of interaction
also shows ghost images anticipating the effect of the operation being performed
Fundamental Concepts in Semiotics 53
Figure 2.6.a
Super Tangrams direct object manipulation for rotations. Screen shot reprinted by courtesy
of Kamran Sedig.
(e.g., the empty frame of a triangle, standing for the result of rotation, the arc cor-
responding to the path traveled by a particular vertex, and handles for changing the
rotation angle). Not surprisingly, DCM interaction promoted better learning than
DOM. Children were far more aware of geometric concepts (because they were
explicitly represented and manipulated in the interface). Yet the verbal elaboration
of the principle, closer to the algebraic formulations at the appropriate level of
abstraction typical of good domain knowledge in geometry was only achieved with
RDCM (Sedig 1998; Sedig, Klawe, and Westron 2001).
The RDCM interface also allows children to manipulate concepts associated with
geometric operations, but they do so in a progressive manner. In the case of trans-
lations, for example, they can start from the most generalized notion of translation
(DCM) and move to the next stage. The interface stops showing the ghost image
of the object being translated. Children have to pay attention to the translation
vectors extremities and their relation to the targeted nal state of the operation. At
the last stage, not only is the ghost image gone but one of the vectors extremities
54 Chapter 2
Figure 2.6.b
Super Tangrams direct concept manipulation interface for rotations. Screen shot reprinted
by courtesy of Kamran Sedig.
is xed to a coordinate. The children then have to pay careful attention to the grid
and think about the length and direction of the translation vector given the result
they want to achieve. Figure 2.7 shows a snapshot of Super Tangrams at the most
challenging stage of RDCM for a translation operation.
Semiotic theory provides an interesting account of Sedigs results (de Souza and
Sedig 2001). DOM, DCM, and RDCM are progressively complex semiotic systems
where the types of representation (with respect to their referent and meaning) have
different levels of efcacy in communicating the principles of Euclidean geometry.
The equivocal perception that direct manipulation interfaces are not good for learn-
ing purposes (Golightly and Gilmore 1997; Holst 1996; Rappin et al. 1997) can be
avoided by an increased degree of semiotic awareness. The problem is not the style
of the interface, but the signs appearing on the interface and the meanings that sign-
maker and sign-taker give to them. Peirces thirdness is the category of mediated
phenomena. Mediation by principles and laws is one of the clearest instances of sig-
nication through linguistic formulation. Therefore, if the nal goal of learning is
Fundamental Concepts in Semiotics 55
Figure 2.7
Super Tangrams reective direct concept manipulation interface for translation. Screen shot
reprinted by courtesy of Kamran Sedig.
to help learners formulate linguistically the rules of geometric translation and rota-
tion in 2-D space, visual representations can never do the whole job themselves.
They can (and do) serve as learning scaffolds for establishing solid interpretations
(building the appropriate meanings) that can (and should) be used in the abstract
formulation of the targeted principles. But symbols form the basis of all discipli-
nary discourse. Therefore, if a given interface style constrains semiosis to indexical
representations of the meanings that one wants to communicate (as was happening
in DOM and DCM), it will not (by itself) support symbolic formulations. RDCM
was more successful than DOM and DCM because the interface included a true
visual language, with the appropriate encoding for angles, axes, vertices, and so on,
as well as for the operations on them. The complex articulation of such signs into
which sentences (operation + parameters) corresponds to an algebraic formula-
tion of the principles involved in translation and rotation.
We can see that a semiotic theory not only provides supplemental explanations
for Sedigs ndings (and supports his claims that DOM is not a problem per se in
software applications for learning purposes), but supports useful generalizations.
The ultimate target of learning is the formation of specic interpretants and the
acquisition of specic signication systems. In Sedigs studies, learners were expected
to master a specic signication system corresponding to a portion of Euclidean
geometry. Other signication systems might refer to (parts of) arithmetic, logic,
physics, biology, history, or poetry. All these disciplines have specic concepts and
principled ways to combine them in order to describe, explain, construct, or dis-
cover other concepts. The elaboration of an appropriate signication system whose
referent is, itself, another signication system (a discipline, or part of it) is a major
semiotic task. The greater a designers semiotic awareness, the greater his chances
to build successful computer applications to support learning. A useful semiotic gen-
eralization is that if one such applications interface does not include signs of the
targeted signication system or does not include an interpreter for valid sign com-
binations in this system, the learners wont be exposed to the ultimate target of
learning.
So the design of learningware should start by a semiotic analysis of the targeted
signication system (corresponding to the domain knowledge) and an exploration
of the interactive and computational possibilities of other signication systems that
can be used to represent the targeted system. Some formal characteristics of signi-
cation systems (e.g., the types and modes of representation they support with
respect to the signs, their objects, and interpretants) will mean that the designer will
56 Chapter 2
have to combine different styles of interaction (as in Sedigs experiment), use a com-
prehensive formal language for interaction (e.g., the language of arithmetic or
algebra used in calculators), or reframe his design target and build a different appli-
cation for every distinct stage along the learning process.
2.4 Pragmatics, Speech Acts, and Culture
Pragmatics and semiotics have intimately related origins, although this is not as
widely recognized as we might expect. Peirce was among the founders of pragma-
tism, a particular philosophical perspective that related the meaning of signs to the
effects they had on other signs (pertaining to ones own individual semiosis, or to
a broader scope of collective semiosis). Pragmatism has assessed more profoundly
than ever before the relevance of [the relation of signs to their users] in under-
standing intellectual activities (Morris 1938, 29). Hence its importance for semi-
otic engineering and its targeted account of how users interact with intellectual
artifacts. The work of pragmatist philosophers gave rise to pragmatics, character-
ized by Morris as a branch of semiotics, whose main goal is to account for the
relation between two specic constituent parts of the sign: representation (repre-
sentamen) and its meaning (interpretant). This goal involves dealing with what
Morris calls the biotic aspects of semiosisthe psychological, the biological, and
the sociological phenomena that inuence interpretation. Notice here a hint to some
misconceived notions, implicit in some HCI approaches to meaning, that we should
be looking for a theory of the relation between representation (representamen) and
referent (object). Such theory would miss the biotic dimensions Morris is talking
about, or mistake them for what they are not. The idea that there might be prag-
matic rules to describe biotic relations, which some pragmaticians and logicians
would be happy to discover, is actually a derivation from one of Peirces late con-
clusionsthat interpretants are a habit, a habitual response to the presence of signs
more or less closely related to their referent. This response can be such that the
objective presence (or even the objective existence) of the referent is irrelevant for
the achievement of the effects associated to the sign. Everyday examples of how
world news broadcast in the media (specic spoken words and selected images)
affects people, markets, and politics serve as supporting evidence for the early prag-
matists claim.
If the relation between sign users and the meanings they produce (with or without
access to objective referents) could be described by rules, human interpretive and
Fundamental Concepts in Semiotics 57
cognitive behavior would be a matter of mechanics. Away from the ethical impli-
cations of a mechanized view of human mind and human culture, this hypothesis
has tempted a number of researchers in articial intelligence (AI), since it carries
the promise of computational tractability (even if partial). If such rules could
be found, we would, for example, be able to build cooperative agents with which
users would interact extensively and productively to achieve a wide range of pur-
poses. But research in pragmatics has brought some bad news for the computa-
tionally inclined. One of the problems with the idea of pragmatic rules is the leap
from the individual to the social scale of meaning-assignment processes. If rules were
to account for the relation between the sign and its interpreter by way of the
meaning he attributes to the sign, what would it mean if I interpreted a sign
in a small store as this store sells hammers and somebody else interpreted it
as this is where hammers are sold? Are we using the same rules? Different
variations of the same rule? Different rules? And how about the interpretations
of other hundreds of customers? How would the ultimate set of rules be general-
ized from individual experience? Or is there no such thing as a truly individual
experience? Are we all just enacting some kind of behavior determined by
universal rules?
One of the most interesting examples of how researchers working with natural
language processing (NLP) in the early days of AI explored the possibilities of using
such rules is Schanks conceptual dependency (Schank 1975). In this perspective,
such different interpretations of as this store sells hammers and this is where
hammers are sold would be related to each other in an associative network of con-
ceptualizations (interpretants) whose relations to each other could be computed in
terms of semantic primitives and case structures. What was not so clear to many
adopters of Schanks conceptual dependency theory is that the meanings encoded
in NLP applications were only a particular preestablished and unchanging semiosic
path. It could at best stand for the knowledge engineering groups particular inter-
pretation of a domain, but certainly not for the semantics or pragmatics of natural
language utterances whose referent belonged to that domain. Communication
between users and systems under these conditions was not only a matter of linguistic
competence but, perhaps more important, a matter of having users restrain them-
selves to following always and only the preestablished semiosic paths a system could
handle. And because this is totally unnatural for any human being, early NLP
dreams have been gradually abandoned.
58 Chapter 2
However, especially for the purposes of HCI, NLP kindled a long-standing philo-
sophical debate: How can any two people understand each other if meaning has
such deep roots in individual experience?
In modern pragmatics and semiotics the answer to this question is often formu-
lated in terms of culture. Brown comments that the function of education, which
extends over many years of our lives, is to expose us repeatedly to the same
structured kinds of experience in such a way that all of us who are exposed to it
develop a common pattern of usage for all meanings involved in those situations
(Brown 1995). Although Brown doesnt mention semiotics in her account of
communicationher focus is on understanding the cognitive basis of language use
her analysis is fully compatible with Peirces views. By and large, it is our repeated
exposure to the kinds of signs that are privileged in our culture and intricately
related to each other that determines the conditions of convergent semiosis, neces-
sary for any two people to communicate. Danesi and Perron (1999) claim that cul-
tures provide the signifying order that affects the path of individual semiosic
processes. But it is Umberto Eco (1976) who has championed the view that culture
and semiotics must be so intimately related that the latter is in fact the logic of the
former.
Pragmatic concepts and theories have inuenced HCI and HCI-related areas. One
of the enduring examples in HCI, especially relative to computer-supported collab-
orative work (CSCW) applications and computer-mediated communication (CMC),
is that of speech acts (Austin 1962; Searle 1969, 1979). The essence of speech act
theory is that language is not only used to represent (perceived) states of reality,
namely, to make statements about what and how the world is or is not, but also,
and perhaps more extensively, to have some effect on this very world, to do things.
It stems from a pragmatist tradition and has evolved into the study of patterns and
principles that speakers adopt to accomplish various actions with words. Speech act
theorists distinguish between different dimensions for analyzing and describing
spoken communication. The distinction between the language users intent (the illo-
cutionary act) and the resulting effect of language use (the perlocutionary act) has
added a considerable degree of nesse to the study of language and communication.
Austin and Searle have both used classications of speech acts in their account of
how words affect the world (or are affected by it). Searles classication has inu-
enced the language-action perspective (LAP) for systems design (Winograd 1987),
which guided the development of The Coordinator (Flores et al. 1988), possibly
Fundamental Concepts in Semiotics 59
the most widely discussed CSCW application to date. The ve basic classes of speech
acts proposed by Searle are

assertives (speech acts that commit the speaker to the truth of what is being said);

directives (speech acts that aim at causing the hearer to do something);

declaratives (speech acts that change the status of the world by virtue of what is
said, by whom and to whom);

commissives (speech acts that commit the speaker to taking some particular course
of action in the future); and

expressive (speech acts that aim at drawing the hearers attention to the speakers
psychological state or attitude).
Popular illustrations of the ve classes of speech acts typically align: assertives to
informative statements; directives to requests and orders; declaratives to (usually
ceremonial or institutionalized) declarations, such as You are now man and wife
when uttered by the celebrant of a religious wedding ceremony; commissives to
promises; and expressives to a wide range of utterances where psychological or
social factors determine and explain a particular use of language (such as in apolo-
gies, reprimands, or praise). Speech act theory has supported the use of computer
systems to coordinate human action and communication. The Coordinator, for
instance, was used for everyday communications in sales, nance, general man-
agement, operations, and planning functions in organizations of a variety of sizes
and types (Winograd 1987, 87). The system viewed each communication as part
of structured patterns of action, aligning illocutionary to perlocutionary acts. Thus,
for instance, if a particular communication was emitted as a request (a directive
act), the system would be able to compute the perlocutionary effect the speaker
expected and thus prompt the hearer to respond to such expectation. Likewise,
assertive and commissive speech acts had a particularly relevant effect in organiza-
tional environments (Flores et al. 1988), given the kinds of commitments implied
by their use in communication.
Not surprisingly, rst-generation systems that adhered so closely to speech act
theory encountered resistance from users. The problem was not with the LAP, but
with the relatively narrow focus on speech acts as interpreted by Searle. By that
time, other pragmatic theorists, such as Leech (1983), for example, had already
broadened the scope of how language is used in communication and shown that,
among other things, politeness and even the principles of irony play a fundamental
role in the success of interpersonal linguistic exchanges. Leechs account of
60 Chapter 2
pragmatics bridges speech act theory and Grices cooperative principle (1975),
whose purpose was to characterize the constraints under which a particular logic
of conversation could be sketched.
The cooperative principle is dened by four maxims, freely paraphrased as
The Maxim of Quantity Participants in a conversation should make their contri-
bution as informative as necessary; not more, not less.
The Maxim of Quality Participants in a conversation should only say what they
honestly believe to be the case; they should acknowledge their doubts about what
they dont know for a fact, and never tell a lie.
The Maxim of Relation Participants in a conversation should only talk about what
is relevant for the ongoing conversation.
The Maxim of Manner Participants in a conversation should express their contri-
bution clearly and unambiguously.
Even a supercial reading of these maxims should allow us to notice that in many
casual types of conversations they are frequently disregarded. For instance, small
talk includes a lot of irrelevant information and is most often not about informa-
tion but about socializing. Nevertheless, the cooperative principle is crucially impor-
tant in conversations for coordinating rational goal-oriented behavior in pair or
group work. This is why it has been used in the design of many knowledge-based
problem-solving systems. Another noteworthy aspect of the cooperative principle is
its descriptive power (which acts somewhat in the opposite direction of the design
situation just mentioned). For example, this principle can account for some popular
HCI guidelines such as Shneidermans (1998) strive for consistency (maxim of
manner) and offer informative feedback (maxim of quantity), or Nielsens (1994)
visibility of system status (maxim of quality) and aesthetic and minimalist
design (all four maxims).
Leech has conceptualized pragmatics in a slightly different way. In his view, prag-
matics is about problem solving. Speakers have goals to achieve (cast as problems
to solve) through the use of language. They exploit a variety of linguistic tools,
which Leech formulates as principles. Unlike rules, principles are regulative: They
apply variably to different contexts of use; they are assessed by more or less rather
than yes or no values; they can conict with each other; and they can be con-
travened without impeding the activity to which they concur (Leech 1983). Prag-
matics is about problem solving because communication requires that language
users make decisions (in a kind of means-ends analysis) about what and how they
Fundamental Concepts in Semiotics 61
are going to communicate in order to fulll their goals. Valued psychological and
social goals, for instance, may create important relevant conicts with one or more
cooperative maxims. Maxims themselves can conict with each other. A simple illus-
tration of such conicts is the role of white lies in sparing the feelings of hearers.
To tell a lie is a frontal violation of the Maxim of Quality. However, this violation
may be less offensive or damaging than the consequences of telling the truth in
various trivial matter situations.
But pragmatic principles dont usually cut across cultures. Leechs own formula-
tion of his politeness principle is an interesting testimony of British culture, more
evident, probably, to an observer from a nonAnglo-Saxon culture. The following
six maxims that constitute Leechs politeness principle have a strong and weak form,
the latter shown in parentheses.
The Tact Maxim (Applicable to Directive and Commissive Speech Acts) When
using language to give orders or make promises, it is polite to minimize the cost to
interlocutors (or to maximize interlocutors benet).
The Generosity Maxim (Applicable to Directive and Commissive Speech Acts)
When using language to give orders or make promises, it is polite to minimize the
speakers benet (or to maximize the speakers cost).
The Approbation Maxim (Applicable to Expressive and Assertive Speech Acts)
When using language to express or state something that affects interlocutors, it is
polite to minimize dispraise of things having to do with them (or to maximize praise
of such things).
The Modesty Maxim (Applicable to Assertive Speech Acts) When using
language to express or state something that affects interlocutors, it is polite to
minimize praise of things having to do with us (or to maximize dispraise of such
things).
The Agreement Maxim (Applicable to Expressive and Assertive Speech Acts)
When using language to make a statement about something that affects interlocu-
tors, it is polite to minimize disagreement (or to maximize agreement).
The Sympathy Maxim (Applicable to Expressive and Assertive Speech Acts) When
using language to make a statement, it is polite to minimize antipathy toward inter-
locutors (or to maximize sympathy toward them).
If we take the modesty maxim, for example, in some cultures maximizing dis-
praise of ones doing (the weak formulation of the maxim, whose strong formula-
tion is minimize praise of ones doing) has the opposite effect. Speakers who insist
62 Chapter 2
on dispraising their good achievements provoke a negative reaction from the hearers
(who interpret the speakers strategy as a scheme to elicit praise from the others).
The same is true for other politeness maxims, which points to the fact that there
probably arent universal pragmatic principles of this sort.
Many tricky challenges in HCI design are thus explained by the inuence of
culture on interpretation and, more important, by the fact that pragmatics is a
matter of principles, not rules. Principles apply more or less to different situa-
tions. Productive conversation results from good decision making, more than from
competent linguistic and domain knowledge. Consequently, HCI design should
emphatically aim at supporting the users decisions about how to interact with
systems and other users. Supporting decisions is not the same as supporting under-
standing and cognition, although the latter undeniably contributes to the former.
One of the clearest distinctions between the two is perhaps that knowledge and
action to support understanding is usually formatted in positive termsexplana-
tions about how to achieve goals, how to carry out tasks and perform operations;
information about the meaning of terms, the function and behavior of objects; and
so on. Knowledge and action to support decisions is usually formatted in compar-
ative, and sometimes even negative, termsanalyses of costs and benets involved
in making choices, troubleshooting hypotheses, instructions for how to diagnose
failure, and so on. Back in the mid-eighties, Lewis and Norman (1986) talked about
designing for errors, an idea that was later condensed into design guidelines about
helping users diagnose and recover from errors quickly and easily. But there is more
involved in users decision making, as I point out in the example that follows.
Qualcomms email software, Eudora 5.0.2, includes a special feature called
Moodwatch to scan incoming and outgoing email for potentially offensive words
and phrases. Users can congure different levels of sensitivity, signaled by a range
of one to three chili peppers (see gure 2.8). Moodwatch has been developed based
on the work of David Kaufer, an English professor at Carnegie Mellon University.
He and Qualcomms spokesman Jeremy James reportedly say that the feature isnt
foolproof, but the English language is complex. Sending the message You are a
moron and complete idiot will rank three chilies. But You are a tiresome buffoon
doesnt raise Moodwatchs hackles one bit (Shankland 2000).
Moodwatch is an attempt to address an important social issue brought about by
the Internetour vulnerability in face of amed verbal attacks coming through
email, chat, discussion lists, or any other kind of computer-mediated communica-
tion technology available today. The solution offered by Qualcomm is based on
Fundamental Concepts in Semiotics 63
lexical and phrasal entries found in a kind of amed speech dictionary of aggres-
sive, demeaning, or rude language, that typically appear in amesor abusive
electronic communications (Delio 2000). Although Moodwatch lacks the kind of
pragmatic competence needed for identifying really aggressive, demeaning, or rude
communication, it helps users avoid inadvertent uses of language that may sound
offensive.
This situation brings up an interesting opportunity for pragmatic decision-support
tools that most of the popular CMC applications havent been extensively explor-
ing to dateonline resources for helping users reach for social and cultural ade-
quacy when addressing hearers in a foreign language. Many users (as speakers) have
doubts about the right way to open and close email exchanges with senior profes-
sionals that they have never met, for instance. Is netiquette the same as etiquette?
What are the dos and donts of politeness in the new media? And etiquette applies
to HCI as well. One of the early versions of the Mac OS Finder offers a good
example of the effect of politeness and cooperation (or lack of either) in designer-
to-user communication. If users inadvertently started typing without having set an
64 Chapter 2
Figure 2.8
Screening amed email in Eudora 5.0.2. Screen shot reprinted by courtesy of Qualcomm
Incorporated.
appropriate active application to receive the input, the system would send a
message saying No one is listening to keystrokes at the moment so you might as
well stop typing. This example shows that, independent of the advent of Internet
and all the recognized interpersonal exchanges that it now affords, HCI design itself
can benet from listening to the concerns of CMC.
2.5 Sign Production, Communication, and Discursive Competence
Although semiotics and communication have a wide intersection in terms of the
general phenomena they investigate, the various theories of communication may end
up having little or no intersection at all with the fundamental semiotic concepts
presented in this chapter. Probably the best known and most frequently invoked
example of how big the differences in perspective can be is Shannon and Weavers
model of communication (1949), which accounts for communicative processes in
terms of probabilistic properties of information source codes and message trans-
mission. No mental, social, or cultural factors intervene in their theory, which has
understandably appealed to engineers engaged in building information retrieval
systems. The ShannonWeaver model, however, has been confusing for those outside
the strict mathematical boundaries within which the theory was proposed. Accord-
ing to Eco, the model can be interpreted as referring to a structural theory of the
statistical properties of an information source; a structural theory of the generative
properties of a source code; a study of how nonsignicant signals are transmitted
through a channel; or a study of how signicant pieces of information are
transmitted for purposes of communication (Eco 1976). The second and fourth
perspectives, for example, are much more useful for semiotics than the rst and
third are.
One of the rst semiotic rereadings of the ShannonWeaver communication
model was proposed by Roman Jakobson (1960), who talked about channels, mes-
sages, senders, receivers, and codes like his predecessors, but in the context of com-
municative functions (including psychological and social). Jakobson structured the
space of communication in terms of six related elements: context, sender, receiver,
message, code, and channel (see gure 2.9). A sender transmits a message to a
receiver through a channel. The message is expressed in a code and refers to a
context. In communication, sender and receiver are alternate roles taken on by inter-
locutors. Although this model is unclear about a number of crucial issues in com-
municationsuch as, for example, whether the context of the message is that of
Fundamental Concepts in Semiotics 65
the senders, the receivers, or bothJakobsons contribution to semiotic studies was
to shed light on how language can be used to draw the hearers attention to certain
elements of this communication model.
Jakobson further dened six functions of language in communication, each cor-
responding to one element of the model. The functions are not mutually exclusive
in a communicative act, which means that a particular message can serve more than
one purpose. The expressive function focuses communication on the sender of the
message; the conative function, on its receiver; the referential function, on its
context; the phatic function, on the channel; the metalinguistic function, on the
messages code; and nally the poetic function, on the message itself (what it says
and how it says it). Semiotic engineering uses this model extensively to structure the
design space of HCI (see chapter 3), based on the examples and correspondences
shown here. Each linguistic example of how the function can be expressed is paired
with an HCI example (gure 2.10af).
Expressive When the speaker says In my humble opinion, this is the best choice,
he is stressing his own attitude vis--vis what he is stating. In the HCI example in
gure 2.10a, notice the use of me and we in the systems message, calling
Eudora 5.0.2 users attention to the senders need for help.
Conative When the speaker says Hi, there. Anybody home?, he is checking to
see if his listener is present and alert in the communicative setting. In the HCI
example in gure 2.10b, notice the SWI Prolog users attempts to make the Prolog
interpreter respond by typing ENTER four times. The sign | is returned, not the
66 Chapter 2
Figure 2.9
Jakobsons model of the communication space.
Fundamental Concepts in Semiotics 67
Figure 2.10.a
Expressive function in Eudora 5.0.2 dialogs. Screen shot reprinted by courtesy of
Qualcomm Incorporated.
Figure 2.10.b
Conative function in SWI Prolog dialogs. Screen shot reprinted by courtesy of Jan
Wielemaker (http://www.swi-prolog.org).
answer to the Apropos command. After repeated unsuccessful calls to the system,
the user realizes that a . was expected. The . makes the system respond to the
users command.
Referential When the speaker says It is cold and windy today, he is directing
the listeners attention to an element of the context when their communication is
taking place. In the HCI example in gure 2.10c, notice the information about CPU
use in Windows 2000, which calls the users attention to details of the machines
state.
Phatic When the speaker says Can you hear me?, he is directing the listeners
attention to the status of the channel (or medium) through which communication
is carried out. In the HCI example that follows, notice that Windows NetMeet-
68 Chapter 2
Figure 2.10.c
Referential function in Windows 2000. Screen shot reprinted by permission from Microsoft
Corporation.
ing users can check the quality of video channel transmission in a separate window
named My Video.
Metalinguistic When the speaker says What does interpretant mean?, he is
directing the listeners attention to the content of the message(s) exchanged during
communication. He is checking the meaning of a message component. In the HCI
example that follows, notice the explanatory message of StuffIt Expander about
what happens to les and archives when they are dragged onto the box.
Fundamental Concepts in Semiotics 69
Figure 2.10.d
Phatic function in Windows NetMeeting 3.0. Screen shot reprinted by permission from
Microsoft Corporation.
70 Chapter 2
Figure 2.10.e
Metalinguistic function in StuffIt Expander. Screen shot reprinted by courtesy of Aladdin
Systems, Inc. StuffIt Expander and StuffIt are registered trademarks.
Figure 2.10.f
Poetic function in Windows Media Player. Screen shot reprinted by permission from
Microsoft Corporation.
Poetic When the speaker says What do you think is the meaning of meaning?,
he is playing with the possibilities of the code used in communication (a play on
words). In the HCI example in gure 2.10f, notice how this Windows Media
Player visualization pattern (a star shape) has been set to go with the music being
played (a Christmas medley).
The interest of transposing Jakobsons functions to an HCI environment is to
explore the communicative dimensions and resources that HCI designers can use to
facilitate their own communication with users and the user-system exchanges. In
each of the previous examples, the function introducing the example is the most
salient. As is the case in linguistic communication, a particular communicative event
may involve more than one of Jakobsons functions. Senders always choose where
they want to place the focus of communication by making one of the functions stand
out more clearly than the others.
Jakobsons model can also help designers understand and make the best use of
some communicative events. For example, the uncooperative and impolite
Macintosh message shows that the system was designed to diagnose correctly a
problem with the communication channel. By saying that No one is listening to
keystrokes at the moment so you might as well stop typing, the system is telling
the user that it somehow knows that the appropriate channel for communication
hasnt been established. Since establishing the channel is a requirement for any suc-
cessful communication, the pragmatically appropriate and expected behavior is that
the system would advance a solution (like asking which of all active receivers, or
active applications, the user wants to address). The system could also anticipate the
problem, as some applications do in certain contexts, by controlling the focus of
interaction in such a way that there is always a receiver for keyboard communica-
tion. Jakobsons model is important because it tells us that all six elements are
involved in any communicative act and that all six functions can be used (in design,
in our case) to prevent or remedy communicative breakdowns.
Whereas Jakobson elaborated on the structure of the communicative space and
the various functions that language can perform in bringing attention to one or
more of the elements in this space, Eco (1976) elaborated on sign functions (not
communicative functions), which he dened as those that correlate elements of an
expression plane to elements of a content plane. The power of this notion lies in
its ability to distinguish between signs and signals, which Jakobson does not set
out to tackle. This distinction allows us to speak of signals as potential elements
of the expression planethat is, elements that have not been systematically or
Fundamental Concepts in Semiotics 71
circumstantially correlated to any other element of the content plane but have the
potential to be. Linguistic examples are easily grasped: for instance, wod is not a
word in English, but it has the potential to be one. wrsit is also not a word in English
but it does not have the phonemic potential to be one (because the sound combi-
nations in it are not part of the English sound patterns).
When it comes to designing (articial) semiotic systems, as is the case in HCI,
it would be helpful to have a theory of sign functions (actual or potential) that
designers could use. Ecos semiotic theory includes a theory of sign production
(TSP), whose goal is precisely to account for all types of sign functions, conven-
tionalized or not, linguistic or not, previously instantiated or not. This last point is
specically important for HCI, given that many of the concepts and contexts that
users have to talk about or interact with in HCI do not have a parallel outside the
computer system where they occur. Inasmuch as the characterization provided by
TSP can help designers of semiotic systems anticipate the quality of communication
supported by the various types of sign functions, this theory can actually advance
a theory of HCI design.
Sign functions are the fulcrum of Ecos theory. Sign functions serve to distinguish
between two major dimensions of studysignication systems and communication
processes. Signication systems are the result of culturally conventionalized sign func-
tions. In other words, whenever culture produces a convention that correlates a certain
set of expressive elements to a certain set of contents, a signication system is in place.
Signication systems constitute the codes available for communication processes.
These processes, however, are not constrained to expressions that belong to a signi-
cation system. Communication processes explore and elaborate on existing signica-
tion systems, but communicators may choose to use innovative or latent sign functions
allowed by signals of the expressive plane that have the potential to participate in sign
functions. The need to use such potential sign functions in communication is well
known in artistic communications, of course, but also in the sort of creative commu-
nication that is required when we want to express a novel concept or idea.
Eco proposes a typology of modes for sign production. Four parameters are used
to distinguish the various modes of sign production (Eco 1976):

the physical labor needed to produce an expression

the type-token ratio of the expression

the type of continuum to be shaped

the mode and complexity of articulation

72 Chapter 2
The physical labor needed to produce an expression corresponds to the cost of
choosing and articulating an element of the expressive plane (e.g., uttering a word,
drawing an image, mimicking a gesture, and so many intentional forms of expres-
sion). Values of this parameter range from recognition, when it sufces to pick a
preexisting conventionalized expression for the intended content, to invention, when
there isnt any preexisting convention to express the intended content. Intermediary
values are, for instance, ostention (the act of bringing up the content from its
original content plane to the expression plane) and replica (the act of producing a
copy of the content in the expression plane).
The type-token ratio of the expression is related but not identical to physical labor.
It refers to the agreement between a particular expressive element (an expression
token) and a class of expressive elements to which it belongs (the expressive type).
Thus, whenever there is an expression system enforced by cultural practices, com-
municating with expressive elements taken from this system is easier than when this
system is not available. The values taken on by this parameter range from simple (or
ratio facilis in Ecos theory) to complex (or ratio difcilis). Complex cases of token-
type expressive relations can be found when the expression type coincides with the
semantics of the content. For example, most maps and diagrams, which are motivated
by the content they convey, are signs expressively more difcult to produce in terms
of token-type ratio because there is no ready-made convention that the sign producer
who is not a professional can resort to. Instead, he or she must observe the determi-
nations of the content and produce an expression token that can be directly related to
its content. With this parameter, Eco (1984) is exploring the idea that expressive con-
ventions (or types) facilitate communication. He also remarks that some intermedi-
ary token-type ratio values appear when, for example, diagrams are conventionalized
(e.g., as a notation). The effectiveness of conventions in decreasing the complexity to
produce diagrams as a means to communicate ideas is proportional to the cultural
basis that supports the convention, often referred to as critical mass.
The type of continuum to be shaped distinguishes two important situations in
sign production. In the rst, the referent or content of expressions is made of the
same material (stuff ) as the expressions themselves. Natural language is abundant
in instances of such homogeneous continui. For example, if we say The past tense
of regular verbs in English is formed by adding an -ed sufx to the innitive form,
the meaning of the expression -ed belongs to the same (linguistic) continuum as
the expression itself. In fact, all of the other signs in the sentence can be argued to
be a case of material homogeneity in sign production. The same is not the case with
Fundamental Concepts in Semiotics 73
other linguistic or pictorial signs like hammer or . These are cases of material
heterogeneity. The interest of analyzing material congruence in communication is
related to self-reference in articial languages. Most programming languages, for
instance, have the ability to process symbols whose referent are symbols in the lan-
guage itself (e.g., and not in a lower-level machine code). This is why programming
languages have been characterized as being naming spaces, among other things
(Gelernter and Jagganathan 1990). When it comes to HCI, all applications where
users can create new symbols in the semiotic systems they use for interaction (e.g.,
be it because of customization, end-user programming, or merely the creation of
objects to which they can refer to while building expressions about other objects)
can benet from an analysis of material congruity between expressions and con-
tents in the semiotic systems.
The mode and complexity of articulation Characterize the degree of grammat-
icalization of the sign being produced. Highly grammaticalized signs are produced
according to complex rules, as is the case of natural language signs, for instance.
Each natural language word follows certain phonemic (relative to sound patterns)
and morphologic (relative to lexical and word-formation patterns) rules. Natural
language sentences, made of words, follow syntactic and semantic rules. And nally
communicative utterances and texts, typically made of phrases and sentences, follow
pragmatic rules, some of which have been mentioned previously. Visual signs, on
the other hand, have a much lower degree of grammaticalization. Icons, for instance,
dont usually have any formation rules to combine their various constituent parts
into a whole (i.e., to articulate them).
Inasmuch as computer science follows the Turing machine model, it becomes
apparent that this parameter is highly productive as a means of characterizing com-
puter programs in general and application interfaces in particular. Many design
guidelines talk about consistency or patterns that designers must follow
throughout an interface project. These notions are typically associated to principles,
much weaker than rules, that provide a sense of unity to the interface. Mullet and
Sano (1995), for example, explicitly invoke a semiotic background to propose design
principles for representation and images. In their own words, when developing
multiple images, care must be taken to maintain cohesion within the image set to
consider the physical, conceptual, and cultural context in which the images will
ultimately be displayed. Mastery of these concepts is far more important than any
innate artistic ability in developing an effective image program (175). For Eco,
the principles followed in this program are a kind of quasi-grammatical code (for
74 Chapter 2
they lack the descriptive and generative power or grammatical rules), an idea
known as pattern languages that has long been used by architects and designers
(Alexander 1977).
Signs may thus be combined in different ways, and at different costs to the sign
producer and consumer. Eco remarks that communication involves a discursive com-
petence, the accumulated knowledge of repeated exposure to certain sign combina-
tions that allows a hearer to know or anticipate what the speaker is going to say
in numerous contexts. Discursive competence has a social dimension, knowledge
that is experienced and reinforced collectively by a group of people, as well as a
psychological one, knowledge that is built individually as a result of repeated expe-
riences and interpretations. Discursive competence plays a major role in communi-
cation, both for success and failure. It contributes to the interpretation of messages
as intertwined contents (Eco 1976), each partially dictated by certain abductive
steps that belong to our discursive competence. In the HCI example that follows,
the trace of discursive competence in determining a users behavior is particularly
illuminating for a semiotic analysis of interface signs.
This HCI scenario tells the story of a user who had to process her email and
wanted to listen to music as she did so. It shows how she made sense of various
interface signs she was exposed to over a period of time, and it provides a power-
ful illustration of semiosis, abductive reasoning, discursive competence, and various
other concepts presented so far.
Fundamental Concepts in Semiotics 75
Joan plans to spend about fteen minutes with email and decides to program a
listening session in Windows 98 CD Player. She wants to listen to Vivaldis The
Winter, which corresponds to the last three tracks in her Vivaldis Four Seasons
CD. Joan doesnt quite remember how to program a listening session using CD
Player because its been a while since she last did this with the help of a friend.
But she thinks it should be easy. Her rst step is to put the CD in the drive, and
it automatically launches CD Player. She pushes the stop button as she listens to
the rst bars of The Spring concerto (in Track 1). She then chooses the option
Edit Play List . . . in the Disc pull-down menu, and a dialog box pops up (see the
right-hand side of gure 2.11). In it Joan sees various text boxes where she can
type in information about her CD, but her attention goes to two list boxes, one
named Play List (on the left-hand side of the dialog box) and another named
Available Tracks (on the right-hand side). Between them there are four push
buttons labeled Add, Remove , Clear All, and Reset. After some struggling
with the interface, she realizes that the listening selection results from removing
the tracks she doesnt want to listen to from the playlist. In gure 2.11 we see
the point where Joan has selected Tracks 1 through 9 (corresponding to the three
concertos she does not want to listen to) and is about to remove them from the
playlist. This is semiotically characterized by the instantiated sign components
in the triangular sign diagram on the left-hand side of gure 2.11. However, she
suddenly hesitates and wonders if that is what she did the last time. The directed
arrow next to the word Remove makes her wonder if those tracks will be
removed from the list on the right-hand side. That she denitely doesnt want.
She does not want the tracks to be removed from her CD!
After some ten seconds of hesitation, Joan decides that her only choice is to
push the Remove button, which she does. To her relief, the playlist on the left
(not on the right, luckily) is shortened to three tracks only: Track 10, Track 11,
and Track 12, which correspond to The Winter concerto. So she closes the dialog
box, presses the Play button on the main window (upper part of the screen on
the right-hand side of gure 2.11), and goes on to process her email.

76 Chapter 2
Figure 2.11
Signs and semiosis while interacting with Windows 98 CD Player. Screen shot reprinted by
permission from Microsoft Corporation.
Discursive competence plays major role in this example. Of course the tracks on
Joans CD have not been erased, but the computer system is breaking an important
principle of this users (and of many others) discursive competence. The topic of
some conversation about a particular computer object does not persist beyond the
temporal limits of the particular conversation when it was rst (or last) introduced.
If this is the interlocutors wish, they must explicitly signal that the topic is to be
resumed. Phrases like as we were saying last time or some other equivalent sign
combination in the communication code are typically used to this end.
This communicative failure on the part of the system adds to Joans equivocal
abduction (motivated by her doubts about the meaning of Remove ), and it leads
her to think that she has positive evidence that she has ruined her CD. The con-
nection between the meanings that the user thought of (in semiosis) but discarded
for pragmatic reasons (her success with programming her rst session) and the ones
appearing much later, on another occasion, supports the view that the scope of inter-
pretation exceeds the limits of particular tasks and situated goals. The interpreta-
tion of all signs is determined by unlimited and ongoing semiosis, which HCI
designers seldom think of spontaneously.
Joan started to have problems when she rst encountered the Remove sign.
She expected the button to be labeled Remove, and not Remove . The directed
arrow, strangely pointing to the list of Available Tracks, posed an interpretation
problem to her. She could not be sure of the meaning of the arrow in the context
of that particular interaction. However, she hypothesized that the arrow was some-
what spurious or irrelevant for that situation. When she pressed the button, she saw
that the playlist looked okay and the playback too. At that point, she concluded
(on the path of abductive reasoning) that the sign was indeed misplaced. It was
meaningless and had nothing to do with the action triggered by the button.
Here again the lack of discursive competence on the part of the system is the crux of
the problem. The arrow does make some sense, but only in a particular conversational

Fundamental Concepts in Semiotics 77

A few weeks later, Joan wants to listen to Vivaldi again, but now she wants
to listen to all of the Four Seasons. So, she picks up her CD, puts it into the
drive, and is surprised to listen to the rst bars of . . . The Winter! Her surprise
turns into panic when she searches next tracks on the CD and nds out that only
Tracks 10, 11 and 12 are scanned. She must have removed the other tracks from
her CD, then!
context when, in the process of programming a session, a user is sending a misselected
track back to the list of available tracks. The effect is actually one of removing the
wrong track from the playlist. The arrow comes, by analogy, from the Add button,
which does seem to transfer (a copy of) a track from one list to the other. But if no
transfer (adding) has been performed by the user in the context of the current discourse,
the presence of the arrow is completely incohesive.
Ecos TSP parameters can also support a semiotic analysis of this interaction (and
might have supported the semiotic design of this interface). The was interpreted
by the user as an articulated sign. Especially in the presence of the next to the
word Add, it suggested to Joan that directionality could be combined with add and
remove actions. But it cant. And she wonders why the arrow was there in the rst
place. At further inspection, Joan might have found out that there is a situation (or
context) in which the arrows make some sense and do stand for directionality. In
gure 2.12, we see that there are two items selected, one in each list, and all buttons

78 Chapter 2
Figure 2.12
The meaning of arrows on Windows 98 CD Player buttons. Screen shot reprinted by per-
mission from Microsoft Corporation.
are enabled. What does add mean in this context? Which of the two items is the
object of the action? The arrows give the users a hint (although of arguable value).
They may be loosely interpreted as after the action is performed, the object it refers
to will be available here. So, Add applies to the item on the right-hand side of
the screen, since the one on the left is already in the location where the arrow points
(to the list on the left-hand side). This reveals yet another articulation problem: The
visualization proposed by this application suggests that there are two focal objects
of discourse, when in fact there should be only one. Items on the right-hand side
can only be the object of the add action, whereas those on the left can only be the
object of the remove action. This is the correct articulation that this visualization
so clearly betrays. A simple token-type ratio for this articulation is to use a single-
focus strategy, as most interfaces do today. Thus, a selection from the Available
Tracks list enables only the Add button, and conversely one from the Play List
enables only the Remove button. Consequently, the correct syntactic combination
could be supported by more rened focus control, instead of additional signs that
have only a limited scope of interpretability.
2.6 Metaphors and Metonymies
Metaphors have been widely explored in HCI as a cognitive resource to help users
understand new concepts by analogy. The historical desktop metaphor introduced
by the Xerox Star station was meant to facilitate the users understanding of le
system actions in terms of everyday life analogs. However, a metaphor is a sophis-
ticated means of expression in natural language (and other codes). It serves various
rhetorical purposes, and it can very effectively enrich the discourse of a speaker,
helping him or her achieve a wealth of pragmatic effects. For example, saying that
semiotics supports a new breed of HCI theories motivates semiosis where such
meanings as cultivate, raise, reproduce, offspring, stock, and others are
very likely to appear. Saying that it supports a new type of HCI theories does not
evoke the same meanings, and the whole idea of a live and dynamic process of
growth is lost. A speaker choosing the form semiotics supports a host of new HCI
theories would have sacriced the idea of homogeneity among the theories (pre-
served by the expressions breed and type) in favor of stressing the quantity of
new theories being talked about. A breed of theories and a host of theories are
metaphorical expressions, although some hearers may not even realize it. Lakoff
and Johnsons pioneer work (1981) in cognitive semantics has shown a wide

Fundamental Concepts in Semiotics 79

spectrum of metaphors we live by and how they correspond to a few primitive
notions that play a fundamental role in cognitive and linguistic competence.
Metaphors are essential for expressing creative thought and naming novel con-
cepts. For instance, an airplanes nose is what English speakers have named the front
part of an aircraft. In the early days of the aviation industry, the words wing, tail,
and others were used in a metaphorical sense that later became a convention (or
dead metaphor). The interesting aspect of these three metaphorsnose, wing, and
tailis that, when it comes to airplanes, the extent of analogy is not the same.
Whereas an airplanes wings and tail have not only a similar (or prototypical) shape
as those of many animals but also a similar function (to y and control the direc-
tion of movement), an airplanes nose has nothing to do with the sense of smell.
Only the shape and location are relevant for the metaphor. This principle allows us
to understand expressions such as the airplanes belly or its eyes, although the
exact path of metaphorical relation is not always the same for all expressions.
Metonymies have historically been treated in association with metaphors
(Aristotle 1984). Whereas the latter create a kind of semantic short circuit between
two concepts (like animals and aircraft in the previous expressions), the former rep-
resent a concept by means of another that is semantically contiguous to it. Fre-
quently used metonymies include representing: the part for the whole, or vice versa;
the cause for the effect, or vice versa; the content for the container, or vice versa;
and so on. For instance, when someone says I like Bach (i.e., Johann Sebastian
Bach), he doesnt mean he likes the person, but his musical compositions. Likewise,
it is literally impossible to drink a glass or two with a meal, but only to drink
some liquid that will be contained in the glass.
Just like metaphors, metonymies serve many important rhetorical purposes. One
of the most relevant ones for our purposes is that of expanding or restricting the focus
of attention of the hearers. An expansion of the focus of attention results in general-
izations like when we say that Europe supported a decision made by the United
States. Neither Europe nor the United States are a person who can make decisions
about this or that. The metonymical rhetoric has the effect of highlighting the unity
and homogeneity of action to the detriment of the diversity of opinions that typically
characterizes a nation or a group of them. When we say or hear this sentence, we may
be aware of but not paying any attention to the fact that not every citizen in Europe
supports a decision that itself is probably not supported by every citizen in the United
States. Our interpretation yields to the focused elements in the metonymical expres-
sion, and we may occasionally mistake gurative for literal meaning.
80 Chapter 2
As was just said, examples of metaphors in HCI have become abundant since
graphical user interfaces were created. The desktop is the most popular example of
how metaphors have been explored to help users understand new concepts in terms
of old known concepts. Metonymies are less frequently discussed, but no less abun-
dant in HCI. Many text editors, like Microsoft Word, for instance, make exten-
sive use of metonymies in editing tasks. Specically, many formatting operations on
words do not require that the whole word be selected. In gure 2.13 we see that
the boldface function is successfully applied to word both when the whole word
is selected before the function is called and when only the cursor is placed inside
the container word. The part (location inside the container) is taken for the whole
(the full content within the container).
The same principle applies in many graphical editors, where operations com-
manded on parts extend to the whole and, interestingly enough, where operations
commanded on a whole grouping of graphical elements (like ll with color) apply
only to the parts that make sense (e.g., polygons are lled with color, but text
fonts are not).
The problem with both metaphors and metonymies is to decide whether they
should be grammaticalizednamely, used as a systematic grammatical construct
in the interface languageor not. Grammaticalized metaphors and metonymies,
unlike what is the case in natural communication, become an organizing principle
of the interface language. In other words, users expect the gure of speech to
extend to all similar contexts. For example, Word users might be confused with the
part-for-whole metonymy in the interactive situations illustrated in gure 2.14,
where some commands apply while others dont.
The effects in gure 2.14 are related to the implementation of the program and
to the hierarchy of objects from a software engineering perspective. Boldface func-
tions apply to words, not to paragraphs; justication functions apply to paragraphs,
not to words. However, the user may of course think of paragraphs written in bold-
face, or of paragraphs aligned to the left or right.
Fundamental Concepts in Semiotics 81
Figure 2.13
How metonymies are used for boldface formatting in Microsoft Word.
This example takes us back to our initial point about how in computation the
difference between langue and parole is lost. Metaphors and metonymies are a
matter of parole, not of langue. However, it is difcult, if not impossible, in any
practical sense, to distinguish parole rules (that are not part of the grammar) from
langue rules (that constitute the grammar) when it comes to specifying computer
languages. This Word example shows that arbitrary decisions about which rules
belong to the grammar and which dont may cause some confusion to users, for
whom metaphors and metonymies are a natural and powerful communicative (and
cognitive) resource.
82 Chapter 2
Figure 2.14
Interactive inconsistencies with metonymical expressions in Microsoft Word.
3
Semiotic Engineering
The goals of semiotic engineering as a theory of HCI are to present an extensive
and distinctive characterization of HCI, to provide a consistent ontology from which
frameworks and models of particular aspects of HCI can be derived, and to spell
out epistemological and methodological constraints and conditions applicable to the
spectrum of research that the theory can be expected to support. This chapter pre-
sents the theoretical tenets of semiotic engineering and sets the basis for the three
chapters of part II, where the theory is applied to evaluating the users experience,
to designing customizable and extensible applications, and to designing multi-user
applications.
3.1 The Semiotic Characterization of HCI
HCI is a particular type of twofold computer-mediated metacommunication. Meta-
communication is technically communication about (aspects of) communication
itself. The most popular aspect explored in metacommunication is how it gives users
a key for interpreting communication itself. Some messages give us a hint of how
their senders mean them to be interpreted, as in the case of Magrittes famous paint-
ing Betrayal of Images. In it an explicit statement saying that Ceci nest pas une
pipe (This is not a pipe) beneath a very realistic painting of a pipe calls the inter-
preters attention to the representational aspects of the image and away from the
referential and cognitive ones.
A similar type of metacommunication is to say that . The message
tells us how to interpret it. Unlike what these two examples may suggest, meta-
communication does not need to be verbal. Nonverbal cues can also be crucial ele-
ments of metacommunication. For instance, the mere presence of a push button in
an electronic document suggests that something will happen if the button is pressed.
In a printed document, however, the meaning of such metacommunication patterns
becomes confusing.
HCI is a specic type of twofold computer-mediated metacommunication in
which computer systems designers send systems users a one-shot message. The
message tells the users how to communicate with it in order to accomplish a certain
range of effects. It is a one-shot message because, from a design point of view,
it conveys a complete and immutable content encoded in and made available by
the systems interface. The message is designed to tell users what can be para-
phrased as follows:
Here is my understanding of who you are, what Ive learned you want or need to do, in
which preferred ways, and why. This is the system that I have therefore designed for you,
and this is the way you can or should use it in order to fulll a range of purposes that fall
within this vision.
The Irst person of discoursein this paraphrase is the system designer (or
a spokesperson for the system design team). The metacommunication is twofold
because it is only fully achieved if users communicate with the message. If they dont,
the metacommunication is not achieved, because the sign standing for the designers
intent is not interpreted as such by the user. We deliberately talk about interpreta-
tion and not understanding because we want to draw attention to the fact that semi-
osic processes (see chapter 2) continuously generate meanings. Some of them may
coincide with the designers, while some may not. Semiotic productivity in HCI is
dictated by the success of designer-to-user metacommunication. As long as users
generate meanings that are compatible with the designers meanings encoded in their
message, user-system communication is successful even if users dont interpret tech-
nology in the same way as designers do.
Three fundamental consequences follow from the semiotic engineering charac-
terization of HCI. First, HCI designers are necessarily involved in the communica-
tive exchanges between users and systems at interaction time. Second, the qualitative
dimensions of HCI that semiotic engineering can help us appreciate must be
anchored in the twofold metacommunication process described earlier. And third,
the process of encoding the designers metacommunication message as a computing
system captures and freezes only a particular segment of the designers semiosis
about the product of his or her design. The actual meaning designers assign to a
system they may have designed is much richer than what is encoded and will
evolve overtime. So will the users meaning. The computationally encoded portion
of the designers meaning constitutes and denes the one-shot metacommunication
84 Chapter 3
message sent to users. It can be generated and interpreted by computational proce-
dures that will manipulate all and only the signs that have been implemented as
symbols in the systems interface languages.
Together, these consequences uniquely determine the design space of HCI and
introduce a new set of dimensions and categories for both design and evaluation of
computer-based interactive artifacts.
3.2 The Semiotic Engineering Design Space
The semiotic engineering design space for HCI follows from how HCI is charac-
terized in this theory. The designer-to-user metacommunication level determines the
user-system communication. The designers message determines both the systems
stance as a sign (that can produce and interpret other signs) and the users role in
communication with the system. In the rst case, the determination results in the
system being a sign that stands for (and refers to) a particular segment of the
designers semiosis. An illustrative example will help us clarify that designers do
mean to say things through the interface.
Arachnophilia 3.9 is an HTML tag-based editor. In gure 3.1 the user has
just pressed the button labeled EditImg (where the cursor still is) and the program
pops up a message box saying that it is good practice to save les before at-
taching images. We can immediately see that this is in itself metacommunication
(since the judgment of what is good practice obviously comes from the systems
designer). Further evidence of how the designer-to-user communication determines
the systems signs (including metacommunicative signs) can be captured in
Arachnophilias online help material, where the systems designer is the rst person
of discourse. He adopts an overtly tutorial attitude, and in the particular case of
introducing an image in an HTML page, this is what he communicates to the users:
Arachnophilia will now ask you whether you want to save your le. This is a very good idea
for several reasons. One reason is that you might otherwise lose your work in the event of
a system crash or power failure. Another is that, if you locate your Web page in a particu-
lar directory, Arachnophilia can automatically move all your resource choices to that direc-
tory also. This makes it very easy to move your completed Web page (or pages) to the Internet
later on.
After you have responded to the Save-File dialog box, hopefully by saving your le,
Arachnophilia will ask you whether you want to automatically copy any les you select to
this same directory. . . . Now navigate to a directory that contains some graphic les and
select one. (Arachnophilia 3.9, 19961998)
Semiotic Engineering 85
This passage provides numerous clues about Arachnophilias designers intent.
Not only does it instruct the user at the operational level (saying how to insert the
image), but it also gives tactical and strategic information about the application
(the sound methods to achieve the insertion and the additional advantages of using
the proposed method). Notice that the designer understands that the user may
choose not to follow his advice (see the phrase hopefully saving your le), which
explains the No button in the pop-up dialog in gure 3.1.
The metacommunication message also affects the way in which the users posi-
tion themselves while communicating with the system. The rest of Arachnophilias
online help material shows that the designer has assumed a clearly tutorial, and even
educational, attitude toward HTML editing. Phrases like Now lets . . .or Now
we . . . express his engagement in promoting the users learning experience. Elo-
quent evidence that users sense and react to this attitude even without resorting to
online help was spontaneously given to us by one of the participants of an exten-
sive evaluation experiment carried out with Arachnophilia (Prates, Souza, and
Barbosa 2000a). He said: This guy must think we are stupid. Why have these huge
buttons in the toolbar? Even without going into the details of who this guy is
(the designer? the system?), this person (a graduate computer science student)
86 Chapter 3
Figure 3.1
Metacommunication in Arachnophilia 3.9. Screen shot reprinted by courtesy of Paul Lutus.
expresses a negative reaction to taking on the user role reected in the interface. In
a post-experiment interview with the evaluator, this user said that the huge buttons
suggested that he (the user) had never seen them before and/or did not have enough
skill to be able to click on smaller ones. And he was annoyed by it.
Thus, the signs emitted by the system have xed implemented semantics (B-
labeled buttons denote activate bold-face function) and algorithmically deter-
mined semiosic paths (the EditImg button in gure 3.1, which triggers the File not
saved dialog). However, the meanings of such signs, for both designer and user,
may (and actually must, according to semiotics) evolve. For instance, after the user
gets a few months of practice, the negative meanings associated with the large
buttons may give way to more functional meanings, probably aligned to a task-cen-
tered semiosis. Likewise, as the designer revisits his decisions, he may come to realize
that the large buttons are an unnecessary exception compared to the rest of the
interface. This evolutionary nature of semiosis points in the direction that there is
no such thing as the meaning of systems signs with respect to either the designer
or the user. However, there is one implemented meaning of these signs, which pertain
to computational processes that interpret them and/or produce them. And although
it may be semiotically vacuous to expect that a particular meaning will be the users
meaning or the designers meaning at any point in time, it is crucially important to
evaluate how a users or designers meaning, contingent on the situation in which
it is examined, compares to the implemented meaning of its corresponding inter-
face sign.
Designing signs that can trigger converging semiosis around its implemented
meaning is the primary goal of HCI design in a semiotic engineering perspective.
This semiotic activity can be structured by means of Jakobsons model (1960) of
communicative functions (see chapter 2). There are six elements to account for in
the construction of metacommunication artifacts in HCI: the sender, the receiver,
the context, the channel, the code, and the message. A designer must decide

what aspects of his/her own constraints, motivations, beliefs, and preferences

should be communicated to the user for the benet of metacommunication (Who
is the sender?);

what aspects of the users constraints, motivations, beliefs, and preferences, as

conceived by the designer, should be communicated to the actual users to support
their projection of themselves onto the role of systems interlocutor (Who is the
receiver?);
Semiotic Engineering 87

what elements of the users expected spectrum of interactive contexts (psycho-

logical, sociocultural, technological, physical, etc.) must be processed by the systems
semiotic computations and how (What is the context of communication?);

what computable codes may or must be used to support effective metacommuni-

cation (including codes that can alternate with each other, codes that are deliber-
ately redundant and should be used in synchrony, codes that complement each other,
codes that supplement each other, etc.) (What is the communication code?);

what channels of communication are available for designer-to-user metacommu-

nication, and how they should or could be used (What is the channel?); and

what he/she (the designer) wants to tell users and to what effect (What is the
message?).
Figure 3.2 sketches the HCI design space structure in semiotic engineering. In it,
the elements of Jakobsons model are identied for correspondence with the pre-
ceding list of decisions a designer must make when producing metacommunication
artifacts for HCI.
Figure 3.2 also expresses the particular interpretation of the computer as medium
concept and, by consequence, the type of computer-mediated human communica-
tion that is involved in my semiotic characterization of HCI. The computer is the
channel through which both the higher-level designer-to-user one-shot message is
conveyed, and the lower-level user-system communication is achieved as part of the
metacommunication process. All messages (at higher and lower levels) are encoded
in computational codes. Some of these messages unfold into other messages, while
others may be atomic. Part of the global context of the designer-to-user metacom-
munication and the user-system communication can and must be encoded in the
88 Chapter 3
Figure 3.2
The HCI design space in semiotic engineering.
one-shot message from designers to users, so that it can play a role in the interpre-
tation and production of signs pertaining to the user-system communication. The
computational encoding of parts (or aspects) of the global context has the effect of
ltering lenses. No matter how rich and dynamic the actual users context may be,
and in which directions it evolves, metacommunication can only picture a xed set
of meanings and semiotic processes encoded in the systems interface. Therefore, not
only is the computer a medium for communication and metacommunication, but
programs in the computer determine the codes, the channels, the messages, the con-
texts, and even the degree of freedom of interlocutors. In other words, we can
expand the formula in many directions, saying that computers are the code, the
channel, the message, the sender, and somehow even (especially in groupware appli-
cations) the user.
The problem with the metacommunication message characterization, so far, is its
apparently passive ontological status and its unclear bond with its legitimate sender,
the designer. Although I have said that the message sends and receives other mes-
sages, it is difcult to see why and when the unfolding communication would take
place, and how it relates to the designers intent. The concept of a designers deputy
completes the ontological underpinnings of HCI as viewed by semiotic engineering.
3.3 The Designers Deputy
Semiotic engineering is committed to the view that software is an intellectual
product. Just as Normans theory of cognitive engineering (1986) underlined the
fact that cognition is central to the users activity, semiotic engineering is now
emphasizing the fact that cognition is also central to the designers activity. Both
application designers and HCI designers are building artifacts that reect their
understanding and reasoning about a number of goals, constraints, opportunities,
and challenges related to using computer technology to affect human life. Addi-
tionally, because the nature of software products is representational, both applica-
tion designers and HCI designers are trying to represent their understanding and
their intent in such a way that the users of their products can see what they mean.
Hence the importance of pragmatics, and what Morris called biotic relations (see
chapter 2), in an account of how users relate to computer artifacts.
The HCI designer is of course more closely involved in making users understand
what he or she means. And, as has been shown so far in this chapter, the way to
accomplish this task is through metacommunication. The designers one-shot
Semiotic Engineering 89
message to users is progressively unfolded and interpreted by the users as they com-
municate with the system. For metacommunication to proceed consistently and
cohesively, the system must speak for the designer. If the system does not speak for
the designer, either the designers message is lost (supposing that users would be
getting somebody elses message through the system) or metacommunication itself
is canceled because some other agent (who is not speaking for the sender of the
designers message) suddenly gets in the way.
The system is thus the designers deputya communicating agent that can tell the
designers message. Because the user communicates with the system, the designers
deputy must of course have elaborate communicative capacities. It must be able to
communicate the contents of the one-shot message, which includes communication
about what it can do.
The communicative (or discursive) competence of the designers deputy must be
analyzed in terms of what it can communicate and how it communicates it. The
object of the designers deputy discourse is exclusively related to the designers one-
shot message. For the sake of coherence and cohesion in the metacommunication
process, no other topic of conversation should be allowed in user-system commu-
nication. Although this may seem to overconstrain the theoretical characterization
of the designers deputy (e.g., think of knowledge-based applications and learning
systems that end up communicating about topics that were not foreseen by the
designers of the original system), at closer examination we can see this must not be
the case. AI applications are designed to learn or acquire predictable kinds of
new knowledge (and no other). Therefore, in spite of not knowing the instances of
new knowledge that an AI application will handle in real use situations, its designer
certainly knows the types of knowledge involved (and how they are represented and
processed by the system). Consequently, the cohesion and coherence of the designers
deputys discourse in communication with the user can be maintained through the
thread of knowledge types and instances.
The form of this communication may vary in many ways, but there are two oppo-
site communicative orientations that roughly correspond to what has been previ-
ously referred to as the model world and the conversation paradigms in HCI
(Hutchins, Hollan, and Norman 1986). The former assumes a reied designers
deputy (i.e., it is presented as a thing), and the latter assumes an anthropomorphized
designers deputy (i.e., it is presented as a human-like agent). Genuine all-reied or
all-anthropomorphized designs are very rare, and they are hardly defensible in
complex and extensive applications where the quality and quantity of tasks is large.
90 Chapter 3
What we see in the vast majority of applications to date is a combination of both.
Typically, some parts of the system reify the designers deputy (which then appears
in the guise of a tool or other physical object), whereas others anthropomorphize
it (and the designers deputy appears in the form of a guide, an expert, or some
other sort of interface agent). The communicative challenges involved in choosing
between the two alternatives center around the designers answer to the following
question: How well will this deputy tell the users what I mean (including what I
mean them to do)? When combining alternatives (which amounts to splitting the
designers deputys role into various specialized communicating entities, reied or
anthropomorphized), the additional challenge is how to alternate them in such a
way that the coherence and cohesion of the one-shot message discourse is not lost.
One of the advantages of introducing the designers deputy as an ontological
element in a theory of HCI is the explanatory power that is gained. The designers
deputy is a missing link in both traditional cognitive theories of HCI and in postcog-
nitive theories as well. In the case of traditional theories, the missing link omits or
rejects the effect of the designers communicative intent. The fact that designers say
that they put affordances in the system is less of a conceptual mistake (Norman
1999) than a strong intuition that the system is a medium for communication with
users (de Souza, Prates, and Carey 2000). Notice that the designers deputy handles
design intent as a distinctive and crucially important part of its communications
with users. In the semiotic engineering perspective, therefore, we do not need to talk
about putting affordances in the interface, but we can certainly talk about the
designers goal of affording certain types of user experiences. And the expression of
this intent can be elaborated and delivered through a variety of codes, depending
on the selected guise of the designers deputy.
Another interesting area in which the designers deputy can play the role of a
missing link is in many theories inspired by anthropology and social psychology
(e.g., activity theory [Kaptelinin, Nardi, and Macauley 1999) or distributed cogni-
tion [Hollan, Hutchins, and Kirsh 2000]). The expansion that these theories have
brought to the scope of analysis supported by traditional cognitive theories is con-
siderable. They provided a rich account of the users experience in the world, where
the computer technology is present, and enlarged the designers horizons about
whom they were designing for and why. However, none of these theories has been
particularly clear about how such expanded horizons translate into design prod-
ucts. The designers deputy offers them a bridge into the nature of computer-based
artifacts and an opportunity to explore the content and form of its interactive
Semiotic Engineering 91
discourse in order to include the dimensions that each theory promotes as
distinctive.
Empirical evidence of the designers deputys presence in HCI is scattered and inci-
dental in many interfaces. Figure 3.1 provided such evidence. But a more common
evidence that someone is speaking on the designers behalf is, of course, online help.
Another is the set of conguration dialogs present in most systems. In gure 3.3 we
see a conguration dialog of the Opera 7.11 browser. Notice how the designers
intent is being communicated when users are told about some particularly interest-
ing things that can be done with the search string (You can even store this sequence
as a bookmark.). Other hints about how well the system knows the user come
from the sentence in parentheses, talking about the place where you normally enter
Internet addresses. Of course it is the designer who knows the user, and the designer
who is calling the users attention to a neat feature of the system. But his or her
92 Chapter 3
Figure 3.3
The designers deputy discourse in Opera 7.11s interface. Screen shot reprinted by cour-
tesy of Opera Software ASA.
discourse about the system and about the user is being conveyed through the
interface.
A full-edged exploration of the designers deputys discourse can only be
achieved if designers consciously recognize it as an element of HCI design. And
because among the theories that have instructed the design of virtually all the exist-
ing interfaces to date none has explicitly included a designers deputy or any equiv-
alent element in its ontology, systematic evidence of how it affects HCI will only
be obtained when semiotic engineering (or any other semiotic theory where the
designers deputy is accounted for) achieves critical mass. Internet applications, and
e-commerce in particular, are very likely to be clients of this type of theory, for pur-
poses of trust. The designer must convey the discourse of the systems owner along
with his own. And we see traces of this explicit rst-person discourse in state-
ments like the one shown in gure 3.4. A condensed combination of trade owners,
systems designers, and HCI designers produced the text offered to customers of
Amazon.com. The opening sentence says that Amazon knows that you care for
how information about you is used. This is the trade owners discourse. Later users
are told to Click here to see some examples. This is the HCI designers discourse.
A couple of lines after that, we see an explanation of cookies and how they are
used to automatically extract information. This is the applications designers dis-
course. The interesting point about this page is that it lets us anticipate the kinds
of theories needed to support this type of design. Since communicating trust is such
an important issue for the success of applications like this one, we need theories
that can bring together all the various speakers who want to talk through the inter-
face into an integrative framework for design. And semiotic engineering is a rst
step in this direction.
One last advantage of theorizing about the designers deputy, before we move on
to the semiotic engineering ontology of HCI, is to clarify some confusing aspects of
the computer as medium approach (Reeves and Nass 1996; Andersen, Holmqvist,
and Jensen 1993). In their work on who users think is the source of communication
when they interact with computers, Sundar and Nass (2000) hypothesized that in the
computer as medium paradigm, users would be talking to a programmer. Although
I do not discuss Sundar and Nasss results here, the point of this reference is to show
that the computer as medium paradigm does not imply that users are talking to pro-
grammers. They are most likely talking to designers (or perhaps the manufacturers
and owners of technology), who know who the targeted users are, carefully analyze
their intents, know their environment and requirements, and produce technology
Semiotic Engineering 93
that can meet the targeted users needs and aspirations. Programmers need not have
the least notion about any of these things. Second, the very idea of the computer as
source, which the study has tested and results have shown to be the favored hypoth-
esis over the computer as medium one, might be rened into a source in the com-
puter paradigm, which is more likely to be the precise case to test. The computer,
in semiotic engineering terms, is the designers deputy, which as we saw may appear
in different guises. By introducing this element in the framework, we can possibly
reconcile one of Sundar and Nasss caveatsthat the participants of their experiment
were computer-literate people who, of course, knew that the computer is not an
autonomous source, although they responded to it in such a way as to exhibit social
behavior that is typical of human interaction. This led them to conclude that a certain
94 Chapter 3
Figure 3.4
Traces of many speakers talking through the Web interface of Amazon.com. Screen shot
reprinted by courtesy of Amazon.com Inc.
anthropomorphism of the computer happened even in the minds of savvy com-
puter users. But the ontological stance of a designers deputy may help us clarify the
situation and proceed into ner distinctions about different representations of the
designers deputy and how people react to each of them.
3.4 A Semiotically Based Ontology of HCI
The sense in which I use the word ontology is not the one proposed by Gruber
the specication of a conceptualization (Gruber 1993). It is closer to the philo-
sophical sense, in which an ontology expresses the categories of things that exist,
from which follows a series of the relations among them. Since our subject matter,
or domain, is specically that of HCI, this ontology will be limited to the categories
of interest for a comprehensive account of the phenomena that semiotic engineer-
ing tries to account for.
Four general categories comprise our ontology: namely, that of signication
processes; that of communication processes; that of interlocutors involved in such
processes; and that of the design space, from which HCI takes its shape. Signica-
tion processes will involve signs and semiosis. Communication processes will involve
intent, content, and expression, as well as two differentiated levels of realization
direct user-system communication and mediated designer-to-user metacommunica-
tion. The interlocutors involved in HCI are designers, systems (the designers
deputies), and users. And nally the design space is characterized in terms of senders,
receivers, contexts, codes, channels, and messages.
The ontology is intentionally minimal, condensing concepts like human, machine,
computer program, culture, grammar, and others, in clusters like user-system
communication or metacommunication. The purpose of this condensation is to
underline some very important epistemological and methodological commitments
made by semiotic engineers and whoever uses this theory to understand more about
HCI or to explore new possibilities in research and practice. The elements covered
by the four categories are interrelated, as will be seen throughout the remainder of
this book. In order to introduce the relations that will be explored in later chap-
ters, I will dene these elements in clusters.
3.4.1 Designer, System, and User
The designer, the system, and the user are the three necessary elements to frame HCI
as a semiotic phenomenon. The majority of implicit or explicit ontologies assumed
Semiotic Engineering 95
by other theories include only the system and the user. But if the designer is excluded
from the root level of the ontology, systems must be postulated as autonomous
signsnamely, signifying entities whose meaning exists exclusively by virtue of the
users semiosis. In this radical user-centered view, the system is devoid of many sig-
nifying dimensions that are intentionally added by the designer. As a consequence,
curious ontological puzzles are encountered when it comes to explaining why people
assign components of human semiosis (such as intent and beliefs) to systems.
What is it about systems, as signs, that cause people to interpret them as if they
belonged to another ontological category, namely, that of human beings? In semi-
otic engineering the system is a crystallization of a particular state (the nal state)
of the designers semiosis with respect to what the users expectations are and what
the users experience should be. And it conveys this crystallized semiosis to the user.
During the process of HCI, the user spots traces of some human semiosis in the
system and reacts accordingly. Although users may not (and typically do not) con-
sciously identify the designer as the source of human semiosis in the system, the
presence of the designer at the root level of an ontology of HCI allows us to theo-
rize about how the system represents and achieves the designers discourse in com-
municating with users.
3.4.2 Sender, Receiver, Message, Code, Channel, and Context
In a communicative process, a sender issues a message in order to achieve certain
kinds of effects. The receiver is the one who gets the senders message. The message
is transmitted through a channel and must be encoded in a signication system that
is shared by sender and receiverthe code. The message always refers to a context,
which is the sum of all elements that affect the semiosis of sender and receiver
involved in the same communication process. As mentioned in section 3.2, all of
these components are causally related in the computer medium. And this has unique
consequences for metacommunication in HCI.
3.4.3 Designer-to-User Metacommunication, User-System Communication, and
Designers Deputy
The designer-to-user metacommunication is a unidirectional and unique (one-shot)
communication process in which the designer (sender) sends to the user (receiver)
a message with the following content: Here is my understanding of who you are,
what Ive learned you want or need to do, in which preferred ways, and why. This
is the system that I have therefore designed for you, and this is the way you can or
96 Chapter 3
should use it in order to fulll a range of purposes that fall within this vision. This
message is encoded in one or more signication systems specially designed to enable
user-system communication and is conveyed through the system (channel or
medium) itself.
The view according to which message and medium collapse into one was popu-
larized by Marshall McLuhans aphorism the medium is the message (McLuhan
[1964] 2002). His argument centered on the fact that each medium requires that
messages sent through it be encoded and decoded in particular ways. In other words,
in his perspective distinct cognitive styles for processing message contents are nec-
essarily associated to different media, and this indissoluble determination of medium
upon message allows us to conceptualize the two as one articulated whole. McLuhan
has been criticized by some for his rather rhetorical formulation of the idea. An
interesting critique came from Umberto Eco (1987), who used Jakobsons commu-
nication schema (the same used by semiotic engineering to structure the HCI design
space) to show that McLuhans evidence for his argument doesnt always refer to
the channel. At times it refers to the code, and at other times it even refers to the
form of the message (its expression). Ecos critique is particularly relevant for semi-
otic engineering because it points directly to how intricately the message (content
and expression), the code, and the channel, in communication, may be associated
to each other (and not only the message and the channel or the medium).
The user-system communication is a lower-level communicative process deter-
mined by higher-level metacommunication. In it a human being (the user, as sender
or receiver) exchanges messages with a computer-based artifact (the system, as
sender or receiver) using a constrained type of articial codesonly those that can
be processed by a computer program. Although the user, as a sender, may intend to
achieve any number of effects and results by communicating with the system, the
system is programmed to achieve only a xed set of effects and results. Likewise,
although the users context for communication may vary freely, the systems
context is structured around a xed set of dimensions and elements that inex-
orably determine the way it processes the users intent, content, and expression. This
type of code ultimately determines the cognitive styles that must be used to process
information and achieve successful communication. Furthermore, because these
codes are themselves determined by the formal symbol-processing capacities of
computer programs (the medium) through which messages are exchanged at all
levels, McLuhans broad statement that the medium is the message nds in HCI
one of its nest pieces of evidence. And Ecos caveat about the mixture of
Semiotic Engineering 97
dimensions that McLuhan seems to be talking about actually unveils the full com-
plexity of HCI.
In a semiotic perspective the designer must be a root-level element in an ontol-
ogy of HCI. But unless there is another ontological element that can channel the
designers metacommunicative intent to the users, there is no cohesive link between
what the designer means to say through or with the system, and what the system
means on the designers behalf. The designers deputy is the seat of the designers
intent and content at interaction time. It is thus a communicative constituent that
legitimizes the system as a sender (i.e., an interlocutor endowed with intentional
states) in user-system communication.
2
3.4.4 Intent, Content, Expression, Signs, and Semiosis
The semiotic theory that provides the foundations for semiotic engineering dis-
criminates between two basic processes: signication and communication. Signi-
cation is the process by which contents are systematically associated with
expressions because of cultural determination. Communication is the process by
which individuals use signication systems and other codes or signs (even ones
that they may have invented) to achieve all sorts of purposes. It follows from this
theory that intent, content, and expression are the fundamental constituents of
communication.
In order to prevent important mistakes in knowledge built from this ontology, it
is crucial to add one more element to itsemiosis. As was seen in chapter 2, semi-
osis is the unlimited sign-production process triggered by the presence of represen-
tations that stand for any quantity or quality of meanings. The process is unlimited
in the sense that no one can fully predict its duration, its path, its content, and so
on. Although some aspects of the process may be motivated (slanting duration, path,
and content, e.g., in one direction rather than the other), the very nature of human
mind and intelligence allows us to take another turn at any moment, and to halt it
or resume at our will. So we see that computer programs are semiotically deprived
of genuine semiosic capacities because semiosis violates the foundational Turing
machine model. According to this model, a symbol-processing machine must obey
a certain set of predetermined rules that dictate all and only the types of interpre-
tations that the machine can assign to symbols. Thus, unlike humans, machine
semiosis is predictable and limited by the rules that govern the machines behav-
ior. What this means in HCI is that the designers deputy can only reproduce a
limited (rule-governed) segment of the designers semiosis. Therefore, in the process
98 Chapter 3
of user-system communication, the designers deputy tells the user only a machine-
processible version of what the designer really meant. And it tells it again and again,
regardless of how the users semiosis is freely evolving around the signs that are pro-
duced in this communication. From a semiotic perspective, this is the most impor-
tant distinctive feature between the H and the C in HCI, one that can explain
communicative breakdowns in human-computer interaction, but alsoand more
importantone that we can and must explore in design in order to expand the types
of signication systems and articial codes that we use to achieve the designer-to-
user metacommunication with success and increase the quality of the users experi-
ence in user-system communication.
3.5 Epistemological Considerations
The commitment of semiotic engineering to the concept of semiosis and its onto-
logical status in a semiotic theory of HCI has very profound epistemological con-
sequences. First, it requires more thorough examination of what we mean by the
users meaning, by the designers meaning, and in truth by meaning altogether. The
meaning of meaning has been the object of philosophical debate for many centuries.
The classic work of Ogden and Richards (1946) contains this illuminating passage:
Normally, whenever we hear anything said we spring spontaneously to an imme-
diate conclusion, namely, that the speaker is referring to what we should be refer-
ring to were we speaking the words ourselves. In some cases this interpretation may
be correct; this will prove to be what he has referred to. But in most discussions
which attempt greater subtleties than could be handled in a gesture language this
will not be so (15).
Since the early days of user-centered design, the goal of HCI design has hinged
on the designers ability to grasp a wide spectrum of the users meanings with respect
to a certain range of purposes and activities that the system should enable. For
instance, the notions of semantic and articulatory distance were proposed by
Hutchins, Hollan, and Norman as a way to measure the cognitive loads imposed
to users by interface languages. The authors dene such distances as follows: Every
expression in the interface language has meaning and form. Semantic distance
reects the relationship between the user intentions and the meaning of the expres-
sions in the interface languages both for input and output. Articulatory distance
reects the relationship between the physical form of an expression in the interac-
tion language and its meaning, again, both for input and output. The easier it is to
Semiotic Engineering 99
go from the form or appearance of the input or output to meaning, the smaller the
articulatory distance (Hutchins, Hollan, and Norman 1986, 100).
We learn from semiotic theory that the users side of these measures is not a
xed point. What the user (as a sender) means by an (input) expression and what
the user (as a receiver) takes an (output) expression to mean is contingent on the
communicative situation in which the expression arises. Although the authors cor-
rectly point out that any interface language has a (xed) meaning (or a xed set of
discriminable meanings) that provides a static reference for measuring semantic and
articulatory distances, it is not clear, either in their theory or in the voluminous work
that has followed the cognitive engineering tradition, how semantic and articula-
tory distances are to be used in HCI design or evaluation. The goal of HCI design
in this cognitive perspective is to build interface languages that are at a small artic-
ulatory and semantic distance from what the users mean. But, if the users meaning
can only be viewed as a transient state in an ongoing semiosic process, strongly
determined by all its previous states, which design goal are we talking about? And
what does it really mean to obtain one or another distance measure in a particular
situation of use?
By embracing semiosis, semiotic engineering gives up the possibility of invoking
the users meaning to measure any denitive quality of HCI design. All measures of
the users meaning refer to the contingencies of the use situation where they were
taken, and as such they point only to the factors that pertain to such contingencies.
The unique advantage of such measures, if compared to similar measures in natural
communication between humans, is that at one of its endsthe systems endthere
is a semiotic agent who is only capable of reproducing exactly the same semiosic
processes in the presence of a predictable and closed set of signsthe signs appear-
ing in the interface language.
The role of culture in human communication is to function as a container of signs
and meanings that cohere in predictable ways into patterns of representation which
individuals and groups can utilize to make or exchange messages (Danesi and
Perron 1999, 67). Such signs and meanings provide a signifying order that can be
traced in individual and group semioses, although they are not the only factors deter-
mining how meaning is produced by individuals and groups, and consequently not
the only factors determining how successful their communication will be. So, in an
HCI context, there are two types of meanings whose distance from each other can
be measured: interface language meanings (because they are xed) and culturally
100 Chapter 3
determined meanings that appear in various segments of the users semiosic
processes.
What semiotic engineering enables us to know then is (a) the complete gram-
matical and semantic specication of the systems interface language (i.e., the
systems signifying competence); (b) the complete specication of how the system
functions as the designers deputy (i.e., the systems communicating competence);
(c) to what culturally determined signs and meanings the systems signifying and
communicating competences are related; and (d) the role that this relation plays in
contingent use situations. So, given what we can know, what should be the goal of
HCI design? That we maximize the ratio of culturally determined signs and mean-
ings in the systems signifying and communicating competence. This will prevent us
from chasing moving targets if we really want to take semantic and articulatory dis-
tances as a basis for design and from misinterpreting contingent distance measures
between the systems and the users meanings as denitive evidence of the appro-
priateness of design.
In view of this reexamination of meaning, our next step in analyzing the episte-
mological commitments assumed by semiotic engineering is to raise a few questions
about preliminary user studies and their role in requirements engineering. Ogden
and Richardss caveat (1946) tells us that if we took sharing the same meaning as
the fundamental criteria for successful human communication, we would possibly
nd out that human communication is, strictly speaking, never successful. To begin
with, there is not a method to elicit the complete meaning of any sign from any
human mind. Consequently, the notion of same should refer to only segments of
the whole meaning. But which segments? Clearly, it is not the case that any
segment will do. Our natural experience with human communication provides us
with abundant examples of how partially shared meanings can lead us to serious
trouble. Pragmatically oriented theories will try to determine the relevant segments
by examining the receivers reaction to the senders message and what can be done
with words (Austin 1962; Clark 1996; Hayakawa 1972; Searle 1979). All segments
affecting the correspondence between the senders intention and the receivers reac-
tion are relevant. But what may seem to be the expected behavior locally from a
receiver in response to the senders message may prove not to be so in a broader
scope. For instance, let us revisit an HCI example presented in chapter 2 to illus-
trate how user and system are not communicating successfully from the beginning,
although the problem only becomes evident further down the interactive path. In
Semiotic Engineering 101
gure 3.5 we recall CD Player, a Windows application that plays CDs and allows
us to program a listening session (see chapter 2). The dialog shown in the gure is
the result of the following sequence of interactions:
1. The user inserts the CD in the CD-ROM drive (users communication).
2. CD Player is launched and displays the number of tracks in the CD, a menu bar
with various options, and a number of push buttons for playback control (systems
communication).
3. The user opens the Disc pull-down menu and selects option Edit Play List (users
communication).
4. The system opens the dialog shown in gure 3.5 with the full list of tracks in
both list boxes (Play List and Available Tracks), and four push buttons for adding,
removing, clearing all, and resetting lists (systems communication).
5. The user selects the tracks that are to be removed from the playlist (users
communication).
102 Chapter 3
Figure 3.5
Programming playlists with Windows 98 CD Player. Screen shot reprinted by permission
from Microsoft Corporation.
6. The system enables the Remove , the Clear All, and the Reset buttons (systems
communication).
7. The user presses Remove (users communication).
8. The system shows the updated playlist on the left-hand side of the screen
(systems communication).
At rst sight, this might seem to be a successful piece of communication, since
the system is doing what the user is asking for. But it may not be so (and often is
not), as the example in chapter 2 shows. The user is not told by the system that the
programming will extend to future listenings of the same CD. So, although we have
a lot of local evidence that user and system are communicating successfully, another
round of communication (days, weeks, months later) will probably show that some-
thing was wrong with this one.
So, going back to the epistemological issue under discussion, although pragmatic
criteria for designating which segments of the users semiosis must be captured
in user studies, the scope of these segments is hard to determine. This example
leads us to an important point about user studies and users requirements analysis:
Designers and analysts have their own semiosis going on before, during, and
after they communicate with each other. So there is no such thing as a complete
understanding of what the users mean, or of what the designers and analysts
mean, when they seem to be talking about the same thing. Consequently, what
seems to be the expected product of the requirements analysis and elicitation
phases in many software development activities to date may in fact be a mirage.
Rather than being a list of features and criteria that will make the system do
what the user wants, it should be taken as a statement of the designers and
the analysts interpretation of which elements of the users semiosis should
be accounted for by the systems semiosis. When they actually come to use the
system, and live with it for a while, users will have different meanings and differ-
ent situations to deal with, of course. And systems should be designed so as to
respond productively and to communicate intelligibly in the presence of constant
change.
A semiotic theory of HCI should then generate the necessary knowledge to help
designers deal with patterns of change. Such patterns would be important in design-
ing error messages, warnings, alerts, online help systems, and the like. All of these
spring from the users action being actually or potentially different from what the
system means to say or do. In early user-centered design terms, we can say that
Semiotic Engineering 103
semiotic engineering should provide the theoretical foundations for designing for
error (Lewis and Norman 1986). But, not only this, patterns of change would also
indicate how the systems semiosis could or should be adapted or extended to
accommodate certain changes in the users semiosis. Theoretical knowledge about
how meaning evolves would then help designers produce systems with appropriate
customization and end-user programming features, for example, two powerful
mechanisms for handling contingencies of use situation in HCI.
The sum of these considerations is that semiotic engineering introduces a focal
shift in what can be known in HCI and what designers can do with it. Whereas
cognitive approaches focus almost exclusively on the users, and on what happens
to them before and during interaction, our semiotic approach focuses on how the
designers semiosis (about the users semiosis) can be crystallized in an interactive
computer system that will communicate productively with users in the widest pos-
sible range of use situations. The types of knowledge pursued by semiotic engi-
neering are then the ones that will identify the cultural underpinnings of semiosis,
the patterns of change that can be represented and computed by means of interface
language processors, the ways in which designers can express their design vision
through computer-based metacommunication artifacts, and nally the ways in
which the users semiosis is affected by the designers metacommunication at inter-
action time. This view is in line with Morriss perception (1938) that semiotics is
an attractive instrument of science, because it forces us to address various aspects
of meanings that may otherwise go unnoticed and undisputed in theoretical
constructs.
3.6 Methodological Considerations
Psychological, anthropological, sociological, ergonomic, and computational theo-
ries should complement and supplement semiotic engineering with specic knowl-
edge about human cognition and emotions, human activity, social structures and
relations in groups and communities, the physical and environmental constraints
for working with computers, and the kinds of symbol-processing machines that can
be built. The main criterion for drawing a line between the contributions of semi-
otic engineering and those that must be complemented or supplemented by other
theories is a methodological one.
First, semiotic engineering should not be used as a predictive theory. We can never
fully predict how the users will get the designers message. Although we can (as we
104 Chapter 3
do in human communication) raise expectations about the effect of certain com-
municative strategies and motivate semiosis along certain paths (and away from
others), there is no way to determine exactly how somebody will interpret and use
signs in communication, except in the case of highly rigid protocols (like login
dialogs, physical interactions with the mouse, and so on) whose scope is too narrow
to be of further interest.
Second, semiotic engineering should explain observable HCI phenomenon. The
basic ontology in the theory contains the elements necessary to structure an expla-
nation for any use situation of interest. The intrinsic quality of the explanation
depends on the internal coherence of the explanatory discourse and on how observ-
able evidence corresponds to the elements of such discourse. The extrinsic quality
of the explanation depends, however, on the amount and the value of new infor-
mation that it adds to explanations provided by other theories.
Third, semiotic engineering should provide the necessary means to formulate HCI
design problems and questions and to elaborate the corresponding solutions and
answers. Because meanings are always evolving, every HCI design problem is a
unique design problem for which unique design solutions must be found. Therefore,
one of the fundamental steps in HCI design is to be able to formulate and refor-
mulate design problems, by asking relevant questions and exploring candidate
answers productively. In a semiotic perspective, design should not be viewed as a
matter of selecting the best matches between a repertory of design problems and a
repertory of design solutions. Repertories of problems and solutions constitute the
culture of design and provide signifying orders and signication systems for every
unique design process.
It follows from this that the methods used in semiotic engineering research should
always enhance the theorys explanatory power and increase the portions of HCI
design culture that can be explicitly used in the elaboration and evaluation of unique
designer-to-user metacommunication problems and solutions. Thus, the kinds of
design tools that this theory is prepared to offer for professional design practice are
essentially epistemic. They should help designers in constantly producing and ana-
lyzing new knowledge that is directly related to design issues. Semiotic engineering
is totally aligned with Donald Schns vision of what the design activity is and
responds to his call for theories that will contribute to an epistemology of prac-
tice (Schn 1983). It provides researchers and designers alike with epistemic tools
to expand their understanding of the nature of HCI and probe the possibilities of
new metacommunication strategies.
Semiotic Engineering 105
3.7 Epistemic Tools for Semiotic Engineering
In Schns view research and design are very similar. Both involve an iterative spi-
raling cycle of analysis and synthesis where knowledge plays the central role.
Researchers and designers always participate in ve major activities: problem
approximation, problem formulation, generation of candidate solutions, evaluation
of candidate solutions, and knowledge reorganization. Knowledge organization
determined by ones own current interpretation of his or her knowledge sources
feeds the other four activities and is itself fed by them. Every time an activity is
successfully completed, it solidies the knowledge that contributed to its success.
And every time an activity is unsuccessful, it challenges the knowledge that was used
to carry it on. Both researchers and designers are experienced practitioners of abduc-
tive reasoning (see chapter 2), although the goals and constraints involved in the
abductive methods they use value different dimensions of the knowledge that is
gained.
Every theory of HCI must contribute to the quality of HCI design products. Semi-
otic engineering has different types of epistemic tools that not only support research
and expand its own frontiers, but also support the knowledge-centered activities in
the design process. They are interpretive models, analytic principles, and analytic
methods. From a research point of view, they all contribute to increasing the
explanatory power of the theory and to synthesizing new epistemic tools. From a
design point of view, they all contribute to naming and framing design problems,
to synthesizing and evaluating solutions, and to elaborating metacommunicative
strategies.
In gure 3.6 we see the main knowledge-based activities that are common to
research and design processes, and where the preceding types of epistemic tools play
a role in the knowledge process. Interpretive models are particularly important in
going from design approximation to design formulation to the generation of solu-
tions. Analytic principles are important in the generation and evaluation of solu-
tions. And analytic methods are especially important for the evaluation of design
solutions. All three types contribute to the (re)organization of knowledge, and all
three relate to the ontological basis of semiotic engineering theory.
In the following chapters, interpretive models and analytical principles and
methods will be illustrated with respect to three different areas of application for
semiotic engineering: communicability evaluation and the organization of the
designers deputy discourse, the semiotic prole of interface and design languages
106 Chapter 3
for end-user programming applications, and the structuring of communicative
processes in multi-user applications.
As will be apparent, semiotic engineering is strongly committed to representa-
tions. They are signs for those who produce them and those who interpret them.
The most obvious signs I have discussed so far are the system itself and all the signs
that appear in the systems interface language (either for input or for output).
However, because the design vision is so crucially important for metacommunica-
tion in semiotic engineering, numerous representations that emerge along the design
process are prime signs in HCI. The theory puts a special emphasis on models and
other related representations of knowledge concepts that are produced and used in
HCI.
In scientic research, models are one of the most useful tools for critiquing and
advancing knowledge. But in design, especially in software design, models are often
thought to be impractical for professional purposes. By adopting Schns perspec-
tive on design, semiotic engineering views models as prime epistemic tools that must,
nevertheless, be made more practical and efcient for the benet of professional
HCI design. Some portions of the models mentioned in the next chapters have been,
and virtually all of them can be, programmed into computer tools to help design-
ers accelerate the modeling process and document various parts of their semiosic
processes. The important implication of the theory for design practice is that the
latter is viewed by the former as a rational activity that continues during HCI itself,
Semiotic Engineering 107
Figure 3.6
Epistemic tools and knowledge activities in research and design.
in smaller or larger scale. It is not that users are or should be eager to know the
design rationale of the artifacts they interact with, because they are not, but that
their semiosis must relate to the designers semiosis, if communicative breakdowns
are to be handled and resolved by systems at all. The ability of a system to
compute the handling and resolution of communicative breakdowns directly
depends on the designers rationalization and representation of design principles,
choices, assumptions, and goals. All of these must be the object of the designers
deputy explicit discourse. As a consequence, semiotic engineering will be an unat-
tractive theory for those who do not or cannot view design as the object of rational
explanatory discourse. But it is likely to be very attractive for those who want
to bring concepts of AI to HCI and work toward the idea of intelligent user
interfaces.
108 Chapter 3
II
Derivation
When your perception only equals that of your teacher,
you lessen the teachers virtue by half.
When your perception goes beyond that of the teacher,
only then can you express the teachers teaching.
Zen saying
4
Communicability and the Designers Deputy
Discourse
There are many desirable -abilities in HCI: usabilitythe most famous, adapt-
ability, applicability, extensibility, exibility, reliability, predictability, sociability,
and others. Communicability is the key quality of interactive computer-based arti-
facts for semiotic engineering. This does not mean that all other -abilities are un-
important or less important than communicability. It only means that, since our
theory is specically equipped to deal with communication and signication, our
focal interest in HCI is the designers ability to communicate the substance and
qualities of their design through efcient and effective interface communication
systems.
Cognitive theories of HCI have claried some of the fundamental purposes of the
kind of communication on which semiotic engineering focuses. For example,
Normans seven-stage model of user-centered HCI (1986) tells us that users have
goals, which they achieve through seven iterated stages: (1) goal formation, (2)
intention to act, (3) planning of actions, (4) physical execution of step(s), (5) per-
ceiving the outcome, (6) interpreting perceptions, and (7) evaluating achievements
in view of the original goal. HCI designers must therefore support users in achiev-
ing their aim by communicating: the range of goals that the system is centrally
designed to achieve; the various methods that can be used to achieve them; the
interface signs that can be used to activate the system to perform the various steps
in a method; and the signs that tell users what the systems response is to their
intervention.
But opportunistic behavior plays an important role in human achievement,
and rational planning is not the only (and not necessarily the best) way of meeting
our ends. Suchman, for instance, suggests that in every different situation we
negotiate steps in the face of possibilities and adapt our plan sketches to the
circumstances around us (Suchman 1987). This point is also raised in the context
of activity theory (Bdker 1989), which calls our attention to the fact that activi-
ties are intrinsically dynamic. They change and evolve all the time, which makes it
impossible to predict exactly how somebody will go about achieving even a cus-
tomary goal. The message for HCI design is that strict plan-based problem-solving
methods may be good for building algorithmic models of intelligent behavior (and
AI applications), but they should not be used to model the users behavior while
interacting with computer artifacts.
A semiotic theory of HCI is particularly attractive because it contains the ingre-
dients to address many important aspects of other theories and approaches. First,
it unveils the fact that all action involved in HCI is primarily communicative. In
other words, all that is achieved in HCI depends on speech acts (see chapter 2)
uttered by the user or by the designers deputy. As a consequence, the users goals,
plans, and activities have by necessity a semiotic extension. They must be cast in
the form of signs. The most fundamental role of the designers deputy is to tell the
user which signs are available for the casting, which meanings they systematically
take on in various interactive situations, and which range of responses to the users
speech acts can be communicated back to the user and how. Second, as a central
concept in the theory, semiosis not only explains but in fact predicts that compu-
tationally encoded meanings pertaining to the signication system in any systems
interface will trigger unlimited interpretation possibilities in the users minds. Some
of these possibilities will coincide with the original designers expectations (and yield
successful results), others will not. The latter are the prime object of HCI evalua-
tion, either because they lead to failure during interaction, or because they point to
productive resignication (a repurposing of the available signs). Interactive failures
(and how to avoid them) are the focus of nearly every method of HCI evaluation.
However, resignication indicates some interesting qualities of software applications
like applicability (to how many different things can it be applied? [Fischer 1998])
and exibility (in how many different ways can the users achieve a particular set of
goals? [Shneiderman 1998]), for example. These qualities open the path to design-
ing and evaluating interactive adequacy in accordance with the criteria favored by
ethnographic studies and activity theory.
This chapter starts by showing how HCI designers can evaluate the communica-
bility of the artifacts they design. It then shows how they can enhance communi-
cability by elaborating the metacommunication discourse conveyed by various
112 Chapter 4
online help functions. The chapter also shows why and how the existence of explicit
underlying models of the artifact (e.g., a user model, a task model, and an interac-
tion model) can improve the users experience from a communicative perspective
(and hence improve it in a number of related dimensions).
4.1 Communicability Dened
Communicability is not a new term for the HCI community. Mullet and Sano have
included it as one of their ve principles for an efcient use of images in the design
of visual interfaces (Mullet and Sano 1995). For them, The communicability of
any representation depends on a shared context between sender and receiver that
allows signs to be interpreted within a pragmatics comparable to the one under
which they were encoded (188). For semiotic engineering, communicability extends
beyond the realm of images. It applies to all the signs included in the various com-
munication codes that constitute the systems interface. In other words, it extends
to the whole signication system that enables HCI within the scope of a particular
application.
In our theory, communicability is the distinctive quality of interactive computer-
based systems that communicate efciently and effectively to users their underlying
design intent and interactive principles (Prates, de Souza, and Barbosa 2000). This
is a metonymic formulation of the concept, which transfers to the design (the
product) a capacity that is in fact expected from the designer (the producer).
The metonymy is commonly used and has appeared, for example, in Brown and
Duguids understanding of the nature and goal of HCI design. See, for instance, the
following passage:
Ultimately, all artifacts have to be interpreted through some codes, which might encompass
such relative but powerful intangibles as attitude, reputation, and intuition. Design must
either try actively to engage in the construction and adoption of codes for interpretation or
else passively accept the codes that users will themselves inevitably construct and engage. We
do not claim that design can determine what codes are used. But it can, like all communi-
cations, attempt to cue relevant sets of codes and to dismiss others. (Brown and Duguid 1992,
170)
Brown and Duguid (1992), for whom tools are cultural artifacts (166), were
discussing semiotic engineering avant la lettre, although they were trying to relate
culture (or cultural codes) to artifact affordances. Communicability has everything
to do with cueing relevant sets of interpretive codes and dismissing others, but it is
Communicability and the Designers Deputy Discourse 113
less about affordances than about discourse. In semiotic terms, the task of HCI
designers is to build a complex system of interactive types according to which users
produce a potentially innite set of interactive tokens. They must build a language
for interaction. And because this language can be made of heterogeneous types of
signs, it is more precisely characterized as an interactive signication system (with
formally denable lexical, syntactic, semantic, and pragmatic components). High
communicability discourse produced in accordance with this system cues a kind of
semiosis (in the users minds) that is compatible with the semantic and pragmatic
denitions of the implemented interface language, whereas low communicability
discourse cannot do it.
Moreover, communicability starts with interpretive codes but extends to expres-
sive ones. Interactive discourse is produced by both the users and the designers
deputy. Both codes are hiding underneath the term affordance. For instance, the
affordance of a sign type like is click & pull down. As part of an inter-
pretive code, users will take it to mean something the designers deputy is saying,
such as have a look at possibilities and choose one (or more). But as part of an
expressive code the same sign type will mean something the users can say them-
selves, such as my choice is 10 or see what Ive chosen. The communicability
of free text entries, for instance, is different from that of pull-down lists. Whereas
in the latter the user is made aware of all existing choices the designers deputy is
able to interpret, in the former the impression he gets (though potentially erroneous)
is that the designers deputy can handle any sign that the user can produce through
the available input devices. This simple illustration shows, additionally, that
although designers cannot truly determine the interpretive codes that users will
adopt during interaction, they can and do determine the expressive codes that users
will employ to communicate with the system.
Communicability can thus be more technically dened as the designers deputy
capacity to achieve full metacommunication, conveying to users the gist of the orig-
inal designers message. This message, as seen in previous chapters, will tell the users
what the designers understanding of who they are is and what they want or need
to do, in which preferred ways, and why. It will also tell them about the system that
the designer has built for them based on what he knows, and how they can or should
use it in order to fulll a range of purposes that fall within the original designers
vision. Communicability applies to both interpretive and expressive codes that the
designers deputy handles for generating and interpreting messages during situated
interaction with users.
114 Chapter 4
4.2 Traces of Communicative Breakdowns during Interaction
No one can predict which is the precise meaning a user ascribes to any particular inter-
face sign. Semiosis is an ongoing sign interpretation process that pragmatically settles
around some meaning conguration when the interpreter no longer has the need, the
impetus, the resources, or the means to continue. This situation poses an important
epistemological challenge for evaluating the communicability of interface signs. When
can we say that high communicability has been reached? Mullet and Sanos answer
when the users interpretation is produced within a pragmatics comparable to the
one under which they were encoded (1995, 188)is correct but of little help if we
think that the set of situations we are talking about is innite. Not even statistical
methods can be safely used when it comes to sampling the innite. Therefore, we need
something else. A simple example of what is at stake is the communicability of the
sign that appears in current Windows applications. At times it will mean close,
at others cancel, or even abort process. But depending on what happens next to
the users clicking on it, it may also mean no harm done or all data will be lost
(some old DOS applications running from Windows clearly capture this clicking event
and warn the users against possible harm to their data). And even when data is lost,
the type of data may be irrelevant to the user (which brings all data will be lost close
to its obvious opposite no harm done). So what is comparable pragmatics? Or,
more precisely stated, what does this criterion mean?
Instances of low communicability, however, are much easier to capture although
the signicance of the problem they are associated with may vary widely (as one
would expect from an innite set of pragmatic mismatches between what the user
means or takes the systems signs to mean, and what is encoded in the underlying
programs). These instances are signaled by numerous patterns of slips, mistakes,
and failures that can be spotted during interaction. So communicability evaluation
starts from examining communicative breakdowns, and from there certain aspects
of communicability, not necessarily negative, are inferred. By necessity, this evalu-
ation can only refer to situated interaction where the users and the designers
deputys discourse can be pragmatically interpreted in view of the actual circum-
stances of their mutual communication.
To illustrate the kinds of communicative breakdowns that semiotic engineering
is prepared to evaluate, let us look at a very simple example of problematic
designer-to-user communication through the designers deputy discourse. The sce-
nario for the illustration is the following:
Communicability and the Designers Deputy Discourse 115
116 Chapter 4
Figure 4.1
Composing messages with Eudora 3.6. Screen shot reprinted by courtesy of Qualcomm
Incorporated.
Joe is using Eudora 3.6 to handle his email. He has a standard signature
that is automatically included at the end of the messages he sends, although he
cannot see the signature text on screen while he composes a message. For
example, the message he is about to send to his boss (see gure 4.1) will include
Joes standard signature, Joe SouzaUsability Engineer ([email protected]
.br), right below the message body. But now that Joe has been appointed Head
of BR-CHI, he wishes to create and use an alternate signature: Joe Souza
Head of BR-CHI ([email protected]). He easily nds out how to create the
alternate signature (see gure 4.2), then sets out to switch signatures for the
message he is about to send to his boss. He wants to sign it as Head of BR-CHI
(new alternate signature) instead of as Usability Engineer (current standard
signature).
In the illustrated narrative, we can follow Joes interaction with Eudora 3.6 Light
closely. The screen shots give the necessary context for interpreting our narrative of
Joes thoughts and actions. His reactions are occasionally expressed by special utter-
ances (Where is it?, I cant do it this way, I can do otherwise, and Looks
ne to me) that are used in the communicability evaluation method presented in
the next section. They characterize a hypothetical response of the user in face of the
designers deputys discourse, and for now it sufces to take them at face value.
Communicability and the Designers Deputy Discourse 117
Figure 4.2a
An attempt to switch signatures using the Tools > Signature menu option. Screen shot
reprinted by courtesy of Qualcomm Incorporated.
Joe goes back to the Tools menu in search of an option to switch his signature
from standard to alternate. The menu options are the same he saw when he
wanted to create an alternate signature, but he thinks that maybe now that he
has already created one, choosing Signature should perhaps lead him to switch-
ing a function. But what he gets from the system is access to the same editing
function he just used to create the alternate signature (see gure 4.2.a).
Oops!
Joe closes the editing window and starts over. He looks for an option switch sig-
natures in other pull-down menus, like Special (see gure 4.2.b).
Where is it?
Since he cannot nd what he wants in the Special menu options, he moves on to
another menu. This time he tries the Message menu (see gure 4.2.c).
118 Chapter 4
Figure 4.2b
Looking for an option to switch signatures in the Special menu. Screen shot reprinted by
courtesy of Qualcomm Incorporated.
Figure 4.2c
Another frustrated attempt in the Message > Change > Status menu. Screen shot reprinted
by courtesy of Qualcomm Incorporated.
There he inspects the Change option, and then Status, but nds nothing useful
there either (see gure 4.2.c).
Where is it?
He looks into various menu options, but still cannot nd it.
I cant do it this way.
He nally decides to change strategy and to frame his problem as a congura-
tion one. So, he explores the Options menu. But . . . no hope there either (see
gure 4.2.d).
I can do otherwise.
Frustrated by useless searches, Joe decides to take a radical step. He opens the
alternate signature le (Tools > Signatures > Alternate), copies the new signature
text, closes the editing window, switches to the message composition window,
and pastes the alternate signature at the end of the message. There goes the
message with a new signature! (see gure 4.2.e).
Communicability and the Designers Deputy Discourse 119
Figure 4.2d
Joe explores Options offered by Eudora. Screen shot reprinted by courtesy of Qualcomm
Incorporated.
The way to change signatures is very simple once you know it: It sufces to select
the value alternate (instead of the default standard) in the pull-down list right above
the message composition canvas (gure 4.2.f). But the communicability of the
designers deputys discourse has some fundamental problems. First, it does not give
the user a clue of what these values refer to. Second, it inexplicably switches the code
type for communicating actions involving standard and alternate signatures.
Whereas you create and edit them through menu selection, you can only switch them
through selecting from a unique pull-down list in the interface. In terms of Umberto
Ecos Theory of Sign Production, presented in chapter 2, this is clearly a case of what
he calls ratio difcilis, or a difcult relation to establish between expression and
content, given the existing codes where this relation is known to hold systematically.
120 Chapter 4
Figure 4.2e
Joe copies and pastes his alternate signature at the end of his message. Screen shot reprinted
by courtesy of Qualcomm Incorporated.
Looks ne to me!
What Joe forgets, however, is that the standard signature is automatically
appended at the end of every message marked to be signed with the standard
signature. So he actually hasnt done what he rst wanted to do.
Later versions of Eudora have a much more elaborate interface for signatures and
personas that the user can create and choose from. Moreover, many usability check-
lists commonly used by designers would have spotted the problem in this interface.
What semiotic engineering adds to the picture is a more perspicuous characteriza-
tion of the problem, and hence a qualitative indication of potential solutions. First,
it shows us the articulation between interpretive and expressive codes in interface
languages. Joes problem was that he could not tell the system that he wanted to
switch signatures. This is an expressive problem. The designers deputy, however,
was telling Joe how he could switch signaturesby activating a particular pull-down
list right above the message composition canvas. But Joe did not get it. This, in its
turn, is an interpretive problem. So, when interpretation and expression dont come
together in communication, there is actually no communication.
Second, in the face of this communicative breakdown, Joe reframes his problem
from one of switching signatures to one of inserting a particular textual content at
the end of a message he is about to send, which tells us his conception of what a
signature is. But the message as-sent is not identical to the message as-seen,
and this indicates to the designer that his sign for the presence of the standard
Communicability and the Designers Deputy Discourse 121
Figure 4.2f
How to switch signatures in Eudora. Screen shot reprinted by courtesy of Qualcomm
Incorporated.
signature (the value standard selected in the pull-down list above the message) is
much weaker than the users sign for the alternative signature (the actual text in it,
at the end of the message). This is a valuable contrast in the (mis)use of signs in the
encoded interface signication system. It gives the designer an indication of prefer-
ential sign types with which to encode the systems feedback to the user. The more
closely the types adhere to the users conception of what the signs mean, the better.
A semiotic analysis of this sort will then tell the designers not only that better feed-
back must be given, but also what is better in this particular case.
Third, the fact that Joe mistakenly thought he had solved his problem (when he
actually hadnt, or at least had not solved it as he rst meant to) points to yet another
problem in communication. Joe doesnt fully master the expressive code he is using
to communicate with the system. He is prone to interpreting the systems signs in
such a way that he nds evidence that his semiosis is in accordance with the encoded
signication system, when it isnt. And in this case, the mismatch is affecting him
(even if he is not aware of it). So the signication system is giving him the oppor-
tunity to strengthen wrong beliefsa dangerous path in HCI. The ability to capture
the problem and to analyze it in detail will give the applications designer the capac-
ity to trigger his own semiosis and think more globally about his design. For
example, the most obvious solution to all of these problems would be what
more recent versions of Eudora actually do: adopt a WYSIWYG (what you see is
what you get) strategy and automatically include the current signature text at
the bottom of the text area where users can compose new messages. In Joes situa-
tion, different problem-framing and problem-solving strategies could emerge if
Eudora followed the WYSIWYG pattern. We might see some of the following
alternatives:

Joe would wish to create the new alternative signature before he started com-
posing the message to his boss. In this case, he would create it with the same kind
of interactive discourse he used in the original scenario, but the designer would have
to help him tell the system that his new message to his boss should be signed with
the alternate signature. A number of possibilities would arise, among them a more
elaborate interaction to start message composition or a default unsigned composi-
tion mode that would always require a specication of which signature to use.

Joe would wish to create the new alternative signature after he started compos-
ing the message to his boss. This would lead the designer to consider more care-
fully interactive discourse regarding message composition. If, for example, the
122 Chapter 4
editing task started with a default unsigned message, the user would always have
to specify explicitly which signature should be used. This is totally impractical if
the vast majority of ones messages uses the same signature (the standard signature,
according to this applications signication system). The very idea of a standard sig-
nature would not make much sense if the effort to use it were the same as that of
using any other signature. But if the editing task started with the standard signa-
ture automatically inserted at the bottom of the text, Joes problem could come up
again. What should he do to replace that text with another? Probably, having the
standard signature in the same text area as the message would lead to complicated
problems: Would it always be possible to automatically delete just the portion of
the text in that area corresponding to the message body? Or should the signature
be in a different text area? Depending on the designers solution, the shortest path
to solving the problem would actually be Joes copy-and-paste strategy, no matter
how clumsy it looked to us in the previous story.
These considerations show that communicability evaluation deals with the users
and the designers semiosis, and alsoas will be seen in the following sections
with the evaluators. It is a prime instrument for reection in action, with the advan-
tage that the categories of objects and phenomena to which it is explicitly related
can guide the designers reection through the design space. These categories emerge
from the semiotic engineering ontology for HCI and are centered around perlocu-
tion or perlocutionary acts (see chapter 2) associated with both the users and the
designers deputys illocution, or illocutionary acts. Major and minor failures in
getting the listener to do exactly what the speaker expects can be distinguished and
analyzed, helping the designers understand and resolve communicative breakdowns
in HCI.
HCI encompasses speech acts performed by the user and the designers deputy.
Both produce illocutions where expression, content, and intent bring about some
effect on the state of affairsperlocution. When perlocution is completely consis-
tent with illocutionthat is, when the effects achieved by what is said fully coin-
cide with what was meant to be the casecommunication is totally successful.
Otherwise, there are various types of problems, implying various degrees of sever-
ity, which can be distinguished according to the discrimination principle presented
here. Global illocution and perlocution refer to the top-level intent of communica-
tion. Local illocution and perlocution refer to lower-level goals, produced on the
way to achieving the original top-level goal.
Communicability and the Designers Deputy Discourse 123
If global illocution is not consistent with global perlocution, then this categorizes
complete failures (I) of which

the user may be aware (Ia)

the user may not be aware (Ib)

If global illocution is consistent with global perlocution but local illocution is not
consistent with local perlocution, then this categorizes temporary failures (II) where

the users semiosis may be momentarily halted because (IIa)

1. he cannot nd the expression to convey the content and intent of his illocution
2. he cannot understand the designers deputy illocution
3. he cannot nd an appropriate intent to compose his own illocution

the user realizes that he must reformulate illocution because (IIb)

1. the context of his illocution was wrong, although his intent was right
2. the expression of his illocution was wrong, although context and intent were
right
3. the intent of his illocutions along a many-step abductive path was wrong

the user studies the designers deputys illocution by (IIc)

1. engaging in implicit metacommunication
2. engaging in explicit metacommunication
3. engaging in autonomous sense making
If local illocution is consistent with local perlocution, then this categorizes partial
failures (III) with

potential residual problems for the user because he does not understand the
designers deputys illocution, and thus somehow fails to do exactly what is expected
(IIIa)

no residual problems for the user because he fully understands the designers
deputy illocution, but somehow fails to do exactly what is expected (IIIb)
What this principle tells us is that there are three main communicative categories
in communicability evaluation: complete failures (I); temporary failures (II); and
partial failures (III). Other important factors notwithstanding, such as the frequency
of breakdowns and the effort required to recover from them, these categories may
correspond to different levels of severity. Complete failures are typically the most
124 Chapter 4
severe ones, whereas partial failures are usually less severe, especially if there are
no residual problems in the users interpretation. Temporary failures fall in between
the two. Subsequent subcategorizations address more specic problems related
to the origin or cause of the breakdown. Ranging from problems with the users
nding the right form to convey valid contents and intents to her being totally unable
to think of a valid intent given the situation, breakdown subcategories can draw
the evaluators attention to some important semiotic characteristics, desirable or
undesirable, that the designers should take into account.
Misunderstandings and disruptions may be related to different aspects of goal-
directed (purposeful) activities. The users semiosis may be mistaken with respect to
the establishment of goals themselves, to the plans she has devised to achieve her
goals, or else to the operations that are needed to carry out plans and achieve goals.
These levels can be characterized, respectively, as strategic, tactical, and operational.
It is interesting to see that some problems at a relatively lower level, such as faulty
operations, can propagate to higher levels. For instance, if the users goal is to create
and edit HTML pages for a small Web site she is about to set up for herself, her
inability to understand how to insert tags or create lists and tables using a particu-
lar HTML editor may lead to different outcomes. The user may give up using that
editor and begin looking for some other tool (e.g., another HTML editor or a generic
text editor that can save text in HTML format). She may also give up her entire
plan (e.g., she may ask somebody else to do the job for her). Yet another possibil-
ity is that the user gets to simulate lists, tables, and other formats. She may, for
example, use indentations and tab spaces, which replicate the same visual effects in
a number of contexts, but bring up problems in others (like when HTML pages are
resized with a browser). Thus, difculties in taking the right actions to perform a
taskor, in our semiotic framing of the problem, difculties in nding an expres-
sion for the illocution to convey valid contents and intentsis a breakdown at the
operational level that may lead to complete failures at the strategic level (the user
discards the alternative to use the editor to produce the desired HTML pages), or
to partial failures at the strategic level (simulated list and table formats are lost if
pages are resized with a browser) derived from the users adoption of the wrong
tactics (simulating formats instead of specifying them correctly). In the latter case,
the residual communicative problem is clearthe user will not realize that she has
misinterpreted the tool until she sees a messy visualization of the pages she created.
And until then, she may have repeated the wrong tactics in a number of other
situations.
Communicability and the Designers Deputy Discourse 125
Such nesse in the analysis of communicative problems is particularly important
for a semiotic approach. By exploring the traces of abductive paths followed by
designers at design time (captured in the designers deputys discourse) and by users
at interaction time (captured in tests and experiments), an evaluator may uncover
crucial qualitative aspects of the signication system constructed by the designer in
view of the one(s) that users actually master and prefer to use. In the next sections
the categorization of communicative breakdowns, as well as their signicance for
HCI evaluation and design, are explained in detail.
4.3 Evaluating Communicability in HCI
The semiotic engineering method for evaluating communicability is based on observ-
ing a number of users purposeful experiences with an application. It concentrates
on analyzing certain (critical) parts of the application. The users behavior during
interaction is analyzed and interpreted with respect to the communicative break-
down categories mentioned earlier. The interpretation can be enriched by specic
mappings and relations between these categories and any number of usability prin-
ciples or guidelines, as well as categories or classes of problems dealt with in other
theories. Evaluation is completed at the semiotic proling stage, with an in-depth
account of the designer-to-user metacommunication. An analysis of interface signi-
cation codes, and of how they are used by the designers deputy and the user to
produce discourse at interaction time, provides the elements that can trigger the
designers semiosis about various possibilities for the current applications (re)design
or for the design of other applications. The communicative potential of certain
interactive design patterns can be used to organize a knowledge base with valuable
practical cases and to encode design culture.
4.3.1 Tagging User-System Communication
The rst step in communicability evaluation is tagging a users interaction with com-
municability utterances. It consists of putting words in the users mouth, in a kind
of reverse protocol analysis. The evaluator watches a playback (or any equivalent
reconstruction) of the users interaction and searches patterns of behavior that cor-
respond to the categories and subcategories mentioned in section 4.2. Such com-
munication problems as the users being unable to interpret some interface element,
not nding where an expected element is, misinterpreting the meaning of perceived
126 Chapter 4
elements, and so on, are paired with one or more communicability utterances,
according to the evaluators semiosis and professional skill. For example, in the pre-
vious Eudora scenario, Joe was mistaken about what the option Alternate in the
Tools menu meant, once an alternate signature was created. He thought that in a
post-creation situation it could mean activate your alternate signature. But it only
meant create or edit alternate signature. The evaluator then puts the utterance
Oops! in Joes mouth when she watches the playback and sees that he realizes
his mistake and backtracks. She tags that particular span of interaction with
Oops! Further along in the scenario, Joe is searching for an option that will allow
him to switch his signature from standard to alternate. Following the same proce-
dure, the evaluator tags this other span of interaction with the utterance Where is
it? The set of basic communicability utterances that can be tagged to interaction
spans is xed and will be presented in detail in section 4.4. Some of them have been
used in the Eudora example, such as I cant do it this way, I can do otherwise,
and Looks ne to me. The outcome of the tagging step is a set of associative pairs
aligning spans of recorded HCI phenomena with communicative utterances (see
gure 4.3).
Tagging is only applicable to goal-directed activities. Therefore, it must be pre-
ceded by a preparation step where the following occurs:

The evaluator studies the application in order to form a preliminary assessment

of the designers metacommunication message.

The evaluator selects crucial portions of the applications that will be used in com-
municability evaluation tests. Crucial portions are typically those where the evalu-
ator senses the possibility of communicability problems. Occasionally the designers
themselves may want to evaluate certain portions of the design in order to gain
deeper understanding about the effects of their choices. Alternatively, such portions
Communicability and the Designers Deputy Discourse 127
Figure 4.3
An association of recorded interaction phenomena and communicability evaluation
utterances.
may also be selected because they have been analyzed using other evaluation
methods, for purposes of comparison or conrmation of various results.

The evaluator prepares use scenarios that will maximally elicit communicative
aspects of HCI in the context of the selected portions of the application, and recruits
a number of participants (usually 3 to 10) that fairly represent the applications
typical user.

The evaluator elaborates questionnaires and/or interview scripts that participants

will be requested to answer before and/or after the test.

The evaluator procures the means to (a) register the participants interaction (e.g.,
using input/output capture software, video cameras, or any other equivalent means);
(b) annotate interaction while she watches the participants activity using a clone-
display, a mirrored window, or plain over-the-shoulder observation; and (c) to
archive the collected data in appropriate medium for multiple reviews and
examinations.
Depending on whether the evaluation is made at early formative stages of design
or later, the nature of communicability breakdowns that can be identied, as well
as the way in which data is collected and interpreted, will vary. For example, using
input/output capture software is impossible if a computational representation of
the application is not available. Again, taking the Eudora example as a basis, if
only a sketch of the various screens were available, then the scenario would
have to be run differently. Instead of clicking and activating the interface as
Joe did, he would probably just indicate to the evaluator what his next step would
be. Therefore, instead of pulling down the Special and Message menus in search of
an option to switch signatures, he would probably just tell the evaluator something
like Then I click here or Then I open this menu. The evaluator would subse-
quently show Joe all the signs that he would successively obtain with each menu.
Joes successive attempts to interpret and use the sketched signs would reproduce
the symptom of a Where is it? utterance, regardless of any actual implementa-
tion of the interface. An interesting aspect of the method when used at this stage is
that, since interaction is reported in words, the test is likely to give the evaluator a
much richer set of signs to work with than what she gets from instrumented labo-
ratory experiments with an actual implemented prototype. Through words, the user
is likely to express certain important signs that are part of his semiosic processes by
saying, for instance, Where do I switch signatures? His utterance in this case not
only explicitly indicates an opportunity to use the Where is it? tag, but also names
128 Chapter 4
the function that the user is looking for switch, but not change or swap,
signatures.
The idea of using utterances such as Where is it? or Looks ne to me
expresses a radical commitment to exploring the facets of communication through
communication. This mechanism advances certain opportunities not only for
improving rsthand exchanges between users and the designers deputy, but also
and just as importantfor using online help resources to improve exchanges about
the problems with rsthand communication. To facilitate this connection, the
current set of communicability utterances has been inspired by research on online
help (e.g., Sellen and Nicol 1990) and on explanation systems (Moore 1995; Cawsey
1993).
Thirteen basic communicability utterances characterize breakdowns in user-
system communication, more appropriately framed as communication between the
user and the designers deputy: Whats this?; Why doesnt it?; Help!; Where
is it?; What now?; What happened?; Oops!; Where am I?; I cant do it
this way; Thanks, but no, thanks; I can do otherwise; Looks ne to me;
and I give up. Basic communicability utterances apply to virtually all kinds of
computer-based technologies. Special technologies like groupware, articial intelli-
gence, and virtual reality, for instance, involve specic types of breakdowns that can
be tagged with Who are you?, Do you know this?, and Let me out, respec-
tively. Such specialized tags have not yet been extensively investigated in semiotic
engineering.
Whats this?
This utterance is used to tag interaction where the user expects to see an explana-
tory tip or any other cue about what a particular interface sign means. The typical
symptom of Whats this? is when the user positions the mouse pointer on an inter-
face sign, waiting for a short explanation to appear on screen (right beneath the
mouse pointer, in a message area, or anywhere else). It may also involve opening
pull-down lists, menus, or dialog boxes to see what they say. The user is probing
the designers deputys illocution (category IIc1). If the user is looking for a specic
sign with which to express his own illocution, it is not a case of Whats this? but
rather of Where is it? Also, if the user explicitly calls a help function, pressing
F1 (as in most Windows applications) or dragging a question mark onto the inter-
face element he is trying to make sense of, the appropriate communicability tagging
is not Whats this? but Help!
Communicability and the Designers Deputy Discourse 129
Why doesnt it?
This utterance is used to tag interaction where the user is trying to make sense of
the designers deputys message by repeating the steps of previous unsuccessful com-
munication in order to nd out what went wrong. The user is probing the designers
deputys illocution by engaging in experimental sense making of his own (category
IIc3). That is, he does not know how to express his intent but suspects that the sign
he is currently examining is the one to be used in a somehow related illocution.
Because of previous unsuccessful experience with the sign he is probing, the user
tries to correct his understanding through repeating the same steps and checking for
potential mistakes. If the user is repeating an operation because he did not perceive
the outcome of the previous iteration and is clueless about what is going on, the
tag to be used is not Why doesnt it? but What happened?
Help!
This utterance is used to tag interaction where the user explicitly resorts to full-
edged metacommunication in order to restore productive interaction. The typical
symptom of Help is when the user deliberately calls a help function by pressing
F1 (as in most Windows applications), dragging the question mark onto the widgets
he is trying to make sense of, or asking somebody for help (e.g., the observer or a
colleague). Reading documentation material ofine also characterizes Help! The
user is probing the designers deputys illocution by directly asking for help (cate-
gory IIc2). As seen previously, if the user is only trying to activate tool tips, without
explicitly invoking the online help function, it is a case of Whats this? not
Help! The motivation for singling out tool tip reading from other patterns of
interaction that yield the same effect (like dragging the question mark icon onto a
specic widget whose meaning one is trying to understand) is twofold. First, an
explicit invocation of help is a different kind of illocution (a direct speech act) than
eliciting information in passing while exploring interactive opportunities (an indi-
rect speech act, considering that a moment of hesitation over the widget is enough
to elicit the information). Second, from a discourse point of view, resorting to help
is the hallmark of metacommunication in HCI. Even if users are not aware of the
designers presence and inuence in interaction while they are involved in perform-
ing tasks and achieving their goals, they are much more aware of the designers dis-
course when reading online help information. This difference in the users degree
of awareness is an important distinction that must be studied and explored if more
sophisticated metacommunication is to be designed. Although used less frequently
130 Chapter 4
than one might expect, online help is a privileged communicative resource for
designers. Later in this chapter I show how communicability and online help design
can be brought together to offer users novel communicative experiences.
Where is it?
This utterance is used to tag interaction where the user expects to see a certain sign
that corresponds to a particular element of his current semiosic process but cannot
nd it among the signs expressed by the designers deputy. His semiosis is tem-
porarily halted, because the user misses the form to convey valid content and intent
(category IIa1). The user must be convinced that the sign he is looking for is the
one he needs to express his current communicative goal. The typical symptom of
Where is it? is when the user is continually browsing menus, pull-down lists, or
any other structured signs. The users search may be guided by his interpretation of
the words or images associated with the structured sign he is currently exploring
(which congures a thematic search) or not (which congures a random search or
exhaustive scanning). If the user nds what he is looking for during short thematic
searches, the breakdown is less severe than if he only nds it after long random
searches or sequential scanning.
What now?
This utterance is used to tag interaction where the user is temporarily clueless about
what to do next (category IIa3). His sense making is temporarily interrupted because
none of the designers deputys signs mean anything to the user. The typical symptom
of What now? is when the user is following a random path in interaction. If there
is a search involved, its a search for opportunity and chance. No goal-directed con-
nection can be traced between one interactive step and the next. The What now?
tag is subtly different from Where is it? especially if, in the latter, the user is
engaged in a sequential search for content. The difference lies in the users knowing
the content he wants to express (the case of Where is it?) or not having a clue
(the case of What now?).
What happened?
This utterance is used to tag interaction where the user repeats an operation because
he cannot see its outcome. In the absence of evidence for the effect of his speech act
(perlocution), the user cannot successfully achieve semiosis triggered by the inter-
face sign he is using to express his intent (category IIa2). As in any sensible con-
versation, the user expects a sign of continuation from the designers deputy. If the
Communicability and the Designers Deputy Discourse 131
users illocution is not followed by the designers deputys, communication is
interrupted. The typical symptom of What happened? is the users repeated acti-
vation of a function whose feedback is either absent or not perceived. Alternatively,
the user may give evidence of noticing a signal (like a icker), but not taking it as
a sign (i.e., not taking it as a meaningful element in the situation). If the user takes
as a mere signal what is meant as a sign by the designer, this is an important com-
municative disruption. If, however, the user is engaged in a fully connected abduc-
tive path (although potentially equivocal) and is fully aware of the signs emitted
by the designers deputy, the tag to be used is Why doesnt it? and not What
happened?
Oops!
This utterance is used to tag interaction where the user momentarily makes a
mistake and immediately realizes his slip. He sees that he used the wrong form in
illocution (category IIb2). The typical symptom of Oops! is when the user is
steadily engaged in a semiosic path and performs an operation that is clearly wrong
or inadequate in that situation. He then immediately backtracks from that step with
an undo function or, if the undo is not available, with any other operation that
will immediately cancel the effects of his slip. The immediacy of the cancellation is
an important factor in distinguishing Oops! from I cant do it this way. Also,
Oops! is an isolated breakdown, an important feature to distinguish it from the
patterns of subsequent unproductive interactions that characterize Where is it?
Typically, Oops! does not involve a search. If it develops into a search for a way
to cancel the effects of a slip, then it indicates a very serious problem in the design.
It means that an incidental mistake can disrupt the users activity so radically that
he has to suspend his original goal and engage all his communicative resources in
restoring the productive path. The problem with the design is often one related to
the designers expressionmainly, its location or the physical representation of its
content (an image or a word). However, the users mistakes may also be fortuitous,
in which case the evaluator must note that this particular tag occurrence is insignif-
icant for communicability evaluation.
Where am I?
This utterance is used to tag interaction where the user is interpreting (and poten-
tially using) signs in the wrong context of the application. The users illocution
would be a valid one in another context (category IIb1). This breakdown is not
132 Chapter 4
unusual in applications with different modes, such as text editors with distinctive
editing and previewing modes. The typical symptom of Where am I? is when the
user tries to execute operations or searches for the signs of one mode while in the
other. The main problem in this breakdown is with the expression of context. The
user is confused by the change in context. Where am I? should not be confused
with Where is it?, although during the users search for an appropriate form of
illocution (the case of Where is it?) the user may occasionally enter a different
mode to see if he nds the expression he is looking for. The evaluators interpreta-
tion of the full interaction span is critically important in distinguishing one case
from the other.
I cant do it this way.
This utterance is used to tag interaction where the user abandons a path of inter-
action (composed of many steps) because he thinks it is not leading him toward his
current goal. The typical symptom of I cant do it this way is when the user sud-
denly interrupts an activity he is engaged in and takes a totally different direction.
It characterizes a realization that his semiosis was wrong and must be revised
(category IIb3). I cant do it this way is different from Oops!, because it implies
a revision of a much longer semiosic path. Whereas in the case of Oops! the user
immediately realizes his mistake and tries to correct it, in the case of I cant do it
this way the user is abandoning (and consequently qualifying it as a revision of
wrong assumptions) a whole line of reasoning and action that corresponds to a plan.
This difference between the operational level of Oops! and the tactical level of
I cant do it this way points to a difference in the severity of the breakdown. I
cant do it this way indicates a breakdown where the user has invested much more
time and cognitive effort in doing the wrong thing than does Oops! The rational
connection between the interactive steps that characterize I cant do it this way
is also crucial to differentiate this kind of breakdown from the one indicated by
What now? Likewise, the subsequent changes of the system state, caused by the
users execution of actions along his path, are the distinguishing feature of I cant
do it this way when compared to Where is it?
Looks ne to me.
This utterance is used to tag interaction where the user is convinced he has achieved
his goal, but in fact hasnt. It is a major communicative failure of which the user is
not aware (category Ib). The typical symptom of Looks ne to me is when the
Communicability and the Designers Deputy Discourse 133
user terminates interaction and the system is in a state that is not consistent with
the users overall intent and goal. On occasion, this phenomenon may occur locally,
as part of a nonterminal interactive plan that the user abandons later. But, this is
not its most relevant indication. In order to illustrate its relevance in terminating
situations, I can mention applications that involve form-lling activities. Some auto-
matically save the information provided by the user as he moves from one form
eld to the next. Others dont. When interacting with an application that requires
an explicit saving operation, if the user thinks that he has successfully nished an
information-providing activity without saving the form, it is a case of Looks ne
to me. There is a mismatch between the systems state (the outcome of the users
last operation before he terminates the activity) and the users understanding of what
this state should be in the case of successful interaction. The Eudora example in this
chapter demonstrates another instance of Looks ne to me. Joe was unaware of
the side effect of his pasting the alternate signature text at the bottom of his message
(the message was sent with two signatures: the pasted one and the standard one).
So he thought the problem was solved (although with a lot of difculty), when in
fact the solution contained an important mistake. The problem is usually related to
the expressions used in the designers deputys discourse. The user does not see
whats wrong and interprets the signs expressing the systems state as an evidence
of success. The actual failure may have different degrees of severity. For example,
it may range from a relatively harmless problem, such as Joes double signature in
a message, to loss of data and other unrecoverable mistakes.
I give up.
This utterance is used to tag interaction where the user explicitly admits his inabil-
ity to achieve his goal (category Ia). Its a most severe problem of complete failure,
whose typical symptom is the users terminating interaction without achieving
success. The distinguishing feature of I give up is the users admitting that he has
run out of some fundamental resource that would allow him to use the application
successfully in a particular situation. Among such resources are knowledge, skill,
time, patience, interest and motivation, and so on. The actual conguration of these
resources is an important aspect for communicability evaluation tests. During the
preparation steps, the evaluator must be careful to decide the quantity and quality
of resources that participants will have for the experiment: how much time, how
much information, the level of commitment with targeted results. All of these have
a profound inuence on human communication in general and consequently on
communicability evaluation as well.
134 Chapter 4
Thanks, but no, thanks.
This utterance is used to tag interaction where the user is fully aware of the
designers deputys discourse regarding the types of conversations that are expected
to lead to a particular perlocution (which coincides with the users intent), but he
chooses to do something different than is expected (category IIIb). It points to one
of the distinctive features of communicability evaluation, because it indicates a very
subtle communicative mismatch that is not captured by other methods (e.g., usabil-
ity evaluation). For example, assume that the designer of the application provides
two ways for the user to terminate interaction: a save and exit and an exit
without saving alternative. The user may be fully aware of the two, but he may
have found out that the operating systems function to terminate tasks can be used
as a substitute for the exit without saving termination dialog. The advantage may
be that the user does not have to perform as many operationsa smaller number
of clicks may be needed, for instance. So, with his own alternative, he saves time.
From a communicative perspective, the designers deputy is inviting the user to
engage in a particular course of conversation, which the user declines because he
has found a more convenient option. The symptom of Thanks, but no, thanks
involves more than simply the use of operating system resources that can somehow
override the applications functions. Some of the applications own functions may
have side effects that lead the system to the same state the user wishes to achieve.
For convenience, the user chooses to decline the conversation that the designers
deputy is expecting to have and follows another pattern of interaction (even if
some risks are involved). This tag indicates an important phenomenon in
communicationthe users preference for something other than what is proposed
by the designers deputy as ideal or, in other words, a mismatch between the
speakers model of the listener and who the listener actually is.
At times the designers deputy may propose a number of alternative paths to the
user. If there is an explicit indication of which alternative is ideal in which situation
and the user chooses to do something else, the user is explicitly declining the
designers deputys advice. If there isnt such explicit indication, but the salience of
signs (which typically corresponds to the effectiveness of the affordances in the inter-
face) suggests that the designers effort to produce and communicate those signs has
been considerable, the user is declining the benets of the designers efforts. Using
the Gricean maxims presented in chapter 2, Thanks, but no, thanks indicates that
most probably the maxims of quantity or relevance have been infringed by the
designer with respect to that particular user.
Communicability and the Designers Deputy Discourse 135
I can do otherwise.
This utterance is used to tag interaction where the user is not fully aware of the
designers deputys discourse regarding the types of conversations that are expected
to lead to a particular perlocution (which coincides with the users intent). He then
chooses to do something different than is expected but achieves the same perlocu-
tion (category IIIa). The crucial distinction between I can do otherwise and
Thanks, but no, thanks is the users awareness and understanding of design solu-
tions. Whereas in the latter the user is fully aware of them, in the case of I can do
otherwise he is not. The typical symptom of I can do otherwise is when the user
achieves his goal (locally or globally) in a way that is not optimal (the most ef-
cient and best t for his situation), or that is unexpected in the presence of the
designers deputys signs in a particular situation. I can do otherwise is usually
(but not necessarily) tagged after some previous breakdowns have occurred. Because
it is related to the users inability to fully understand the design solution, it is likely
that the nonstandard solution will be the outcome of the users attempts to nd the
standard one. However, I can do otherwise may also indicate a residual miscon-
ception of the user with respect to how the application works (a more serious
problem). For instance, the user may be systematically performing some task (e.g.,
exit without saving) without knowing the potential problems that he may
encounter if only minor changes in the situation apply. Suppose that the side effect
of using the operating systems resources to interrupt processing, unlike that of the
designers deputys suggestion to exit without saving, is the automatic generation
of a temporary backup copy. The copy may be stored in a temporary directory and
be prompted for recovery the next time the application is used. The user may assume
that the opening conversation asking him if he wants to recover the backup copy
is some irrelevant interface babble, and he may get into the habit of just clicking
on no without really bothering to understand what it means. This is an obvious
opportunity for major problems in the future, when knowing what backup copies
are, what they must be used for, and the resources required to generate and store
them will be the difference between relief and disaster.
The value of paying attention to instances of I can do otherwise is to direct the
designers attention to residual meanings, especially when equivocal. These mean-
ings become an active part of the users semiosic processes whenever he is interact-
ing with the application, and they can be used to explain a number of problems
along the users learning curve. Capturing this kind of breakdown, which may go
136 Chapter 4
unnoticed with evaluation methods that take success at face value (e.g., measuring
only user satisfaction and mismatches between the meanings of expected and
achieved goal states) is advantageous for designing computer-mediated communi-
cation. Among other things, it bypasses the troublesome concept of a mistake
when the user achieves his intent in unexpected ways, but it retains the trace of a
communicative mismatch. It also allows for ne distinctions between cognitive and
communicative aspects. It will handle situations in which the user has enough cog-
nitive skills about goals, plans, and operations, but engages in problematic com-
municative alternatives (e.g., ignoring the perlocutionary effects of suboptimal
illocution).
The thirteen communicability utterances are in one-to-one correspondence with
the communicative breakdown categories and subcategories presented earlier. This
correspondence is shown in table 4.1.
4.3.2 Interpreting Tagged User-System Communication Logs
The interpretation of tagged interactions is directly dependent on the evaluators
semiotic awareness and skill. The categorization of utterances per se provides impor-
tant input for her analysis of what is going wrong (and right) with the application
under study. However, the greater her familiarity with the concepts presented in
chapter 2, the more productive her analysis. Just as important as watching the inter-
actions playback is the examination of what the annotations made during the test
and the pre- and post-interviews say. Post-test interviews are particularly important
to disambiguate situations where more than one tag could be used, depending on
what the user had in mind. For instance, the difference between a Where is it?
and a What now? depends on the users having established the content and intent
of his next illocution or not. The evaluator cannot always tell this difference from
just watching the interaction playback, and often not even from attentively follow-
ing the users illocutions at interaction time. She must identify such ambiguities
during the test, and she must interview the user afterward to have the most exact
picture possible of the users semiosic states throughout the experiment.
Communicative success is marked by the absence (or insignicant amount) of
communicative breakdowns. And communicability problems primarily depend on
the following factors:

the frequency and context of occurrence of each type of tag;

the existence of patterned sequences of tag types;

Communicability and the Designers Deputy Discourse 137
138 Chapter 4
Table 4.1
Categorization of communicability utterances for tagging observable interaction.
Category
type Subcategory Distinctive feature Utterance
(I) (a) User is conscious of failure. I give up.
Complete (b) User is unconscious of failure. Looks ne to me.
Failures
(II) a. Users (1) because he cannot nd the Where is it?
Temporary semiosis is appropriate expression for his
Failures temporarily illocution
halted (2) because he does not perceive What happened?
or understand the designers
deputys illocution
(3) because he cannot nd an What now?
appropriate intent for
illocution
b. User realizes (1) because it is uttered in the Where am I?
his illocution is wrong context
wrong (2) because the expression in Oops!
illocution is wrong
(3) because a many-step I cant do it this
conversation has not caused way.
the desired effects
c. User seeks (1) through implicit Whats this?
to clarify the metacommunication
designers (2) through explicit Help!
deputys metacommunication
illocution (3) through autonomous sense Why doesnt it?
making
(III) (a) User does not understand the I can do otherwise.
Partial design solution.
Failures (b) User understands the design Thanks, but no,
solution. thanks.

the level of goal-related problems signaled by the occurrence of tag types and
sequences; and

the subsidiary ontologies or classes of HCI problems from other theories,

approaches, and techniques, with which the evaluator has enriched her
interpretation.
The frequency of occurrence of each tag is obviously indicative of qualitative
aspects of communicative breakdowns. For example, a high frequency of Oops!
tags probably indicates ambiguities in the designers deputys illocution. A high fre-
quency of Whats this? indicates a mismatch between the signication systems
with which the user is familiar and those used by the designer to express the
designers deputys discourse. A high frequency of Thanks, but no, thanks indi-
cates a mismatch in user modeling, if the participants have been carefully selected
from among the targeted population of users. If not, it may indicate a problem with
the test itself. A high frequency of Thanks, but no, thanks is nevertheless prefer-
able to a high frequency of I can do otherwise, which indicates that the user is
not getting the designers message or is getting it in the wrong way (with potential
residual problems for future interaction).
The frequency of tag occurrence along the users learning curve is likely to change
(Prates, Barbosa, and de Souza 2000). As the users familiarity with the application
increases, tags like Where is it? or Whats this? tend to become less frequent
within the same range of perlocutions. An increase in the frequency of occurrence
of Thanks, but no, thanks typically indicates the development of an interactive
idiolect (i.e., a personal way to use the signication system to achieve communica-
tion). This may be used to help the designers explore alternatives for letting the user
customize the application, or to elaborate different styles of interaction for specic
classes of users. A persistent frequency of occurrence of I can do otherwise may,
however, be troubling. It may mean that the user is repeatedly getting some aspect
of the designers deputys message in the wrong way and potentially reinforcing his
semiosis around an equivocal model of the application.
The context of occurrence of each tag is equally important. The iterated occur-
rence of Where am I? tags in the same context will certainly contain clues of a
preferential type of abduction that has not been taken care of by the designer. For
instance, in Pine, a mail tool for Unix and Linux shells, the sign I is used to
mean show index for current mail folder. The perlocution of saying I as soon
as Pine is launched, for most users, is the visualization of the users Inbox contents.
But if users frequently say I to view the attachments of a message (whose
Communicability and the Designers Deputy Discourse 139
appropriate illocution is > or V), for instance, it is a noteworthy case of Where
am I? An attentive analyst will spot in it an invariant problem in the users semi-
osis. I, for such users, means something like show me the contents of [it]. But
the designers deputy interprets it as show the index of the current folder. There-
fore, although I is a perfectly valid utterance from the users perspective, if he
means to see the contents of [the current messages attachment], it is not so for
the designers deputy. The user is transferring an invariant meaning to a different
context, something that the designers deputy is not prepared to handle. But why
not? I doesnt mean anything else when a message is open. And if it did, it would
probably be the likely cause of many Oops! tags. So the tagging and its context,
in this hypothetical case, would actually be telling the designers not only the cause
of the problems, but also how to eliminate them.
At times, certain patterned sequences of breakdowns may occur repeatedly in the
same experiment. Upon further examination, the pattern may tell the evaluator that
the users semiosis is not being appropriately cued. For instance, a patterned
sequence of Where is it? followed by I can do otherwise suggests that the user
tries to nd the right thing to say, but, having failed, he adopts a suboptimal solu-
tion. The user misses this privileged solution, which must be frustrating for the
designer. The I can do otherwise may be more a way out of the problem than a
proper a solution. And, if so, the user may be also very frustrated with the experi-
ence. The importance of the pattern is that this situation most probably means some-
thing different than an I can do otherwise that occurs independent of previous
breakdowns. In this other pattern, the user may think that his illocution is the right
thing to say given the situation. And although this may be just as frustrating to the
designer (because the user missed what he was telling him), the user may be satis-
ed and convinced that he understands the application well.
Likewise, patterns of long versus short sequences of such tags as Where is it?,
Whats this?, Oops!, and others mean different things. Typically, a sequence of
breakdowns of the same type means that the user is having great trouble not only
restoring productive communication, but also nding signs that will change the
direction of his semiosis and take him out of the problematic loop.
The level of goal-related breakdowns is also highly informative. In particular, it
may indicate the case of some usability problems such as low productivity and user
satisfaction (which belong to categories of HCI problems that are foreign to the
semiotic engineering ontology). For example, the designer of an application may
have designed solution paths that facilitate the users tasks and considerably shorten
140 Chapter 4
the time for completing an activity. This may be one of the advantages marketed
by the applications vendors (or something strongly related to it). If the users get
the designers message and adopt the suggested tactics to meet their typical goal(s),
the benet of using the application will be clear (and commercial success probably
near). However, if they dont get the designers message at this tactical level, all the
pluses will be potentially compromised. Conveying such messages depends on
successfully conveying operational-level messages. The added value of a semiotic
analysis is the realization that lower-level communicative breakdowns may cause
upper-level breakdowns. An analysis of causes and patterns of lower-level break-
downs may indicate some alternatives. For example, users may be missing the tac-
tical-level message because there is a step in it during which their semiosis is
temporarily halted. And how they restore communication after this breakdown is
typically a trace of how the main cue for the designers privileged tactics was
lost. That is, users nd a way to reconstruct meaning around a less efcient tacti-
cal alternative. A reinforcement of this suboptimal solution (i.e., learning that the
right thing to do is to follow the path that eventually led to success) characterizes
a very important communicative problem for designers. The users may nd that the
application does not deliver the promised gains in productivity (which is not true,
in fact). The advantage of communicability evaluation is the possibility of nding
out the causes of halted semiosis. The solution may even be very simple. A change
of signs in the designers deputys discourse, ranging from adopting a totally
different signication system to adopting only a different rhetoric in discourse pro-
duced with the same signication system, may eliminate the problem. The traces of
the users semiosic processes, collected during communicability evaluation tests, will
be a most valuable input to the designers decision. Finding that users are missing
tactical affordances because they cant make sense of the operations involved in it,
or because they cant command these operations appropriately, is a distinguished
result of semiotic engineering evaluations of HCI (de Souza, Prates, and Carey
2000).
The evaluators analysis and interpretation may be considerably enriched if the
semiotic phenomena derived directly from tagging and communicative breakdown
categorizations are paired with factors and dimensions accounted for by other the-
ories, approaches, and techniques. For example, the cognitive walkthrough method
(Wharton et al. 1994) is, as its name suggests, strongly committed to cognitive the-
ories of HCI. It can thoroughly characterize how easy (or difcult) it is to learn the
application from interaction alone. This cognitive (learning) process is equivalent
Communicability and the Designers Deputy Discourse 141
to sense making (a semiotic process), which suggests that many of the ndings result-
ing from a cognitive walkthrough probably have counterparts among the results of
communicability evaluation. In table 4.2, an alignment of the main issues dealt with
in cognitive walkthrough evaluation and those in communicability evaluation is
presented.
It is noteworthy that some of the breakdown categories dealt with in communi-
cability evaluation have no direct correspondence with the general classes of issues
raised by the cognitive walkthrough method. The category corresponding to the
Thanks, but no, thanks tag, for instance, poses an interesting challenge for cog-
nitive approaches in general, and for cognitive walkthrough in particular. How will
an evaluator predict that users will decline the designers privileged solution and
choose another alternative? This epistemic limitation, derived from the foundations
of the method itself, suggests that evaluation methods guided by traditional task
models (where alternative plans to achieve a xed set of predicted goals are speci-
ed) are likely to produce larger-grain characterizations of the users experience.
Typically, they will concentrate on goal-related levels of problems and delegate the
nesses that are explicitly dealt with in a semiotic engineering categorization of com-
municative breakdowns to the evaluators ability to relate problems to uncatego-
rized dimensions of the design. Practical instructions about how to use the cognitive
142 Chapter 4
Table 4.2
Pairing potential HCI problems dealt with in the cognitive walkthrough and communicabil-
ity evaluation methods.
Communicability breakdown categories
Cognitive walkthrough issues corresponding to the following tags
Will the correct action be evident to the user? What now?
(Will the user know what to do?) Whats this?
Help!
Will the user notice that the correct action is Where is it?
available? (Will the users see the systems cue?) What happened?
Whats this?
Help!
Will the user interpret the systems reaction What happened?
correctly? Whats this?
Help!
Why doesnt it?
Looks ne to me.
walkthrough method give explicit indications of the evaluators skill in gathering
critical information for (re)design. Preece, Rogers, and Sharp (2002), for instance,
say that:
As the walkthrough is being done, a record of critical information is compiled in which:
The assumptions about what could cause problems and why are recorded. This involves
explaining why users would face difculties.
Notes about side issues and design changes are made.
A summary of the results is compiled. (P. 421)
The cause of goal-related problems, as seen earlier, is often found in communi-
cation. Therefore, a categorization of communicative problems will leverage the
evaluators diagnostics. Likewise, side issues from a cognitive perspective, are
often central issues from a semiotic one. The case of Thanks, but no, thanks
tags is an example. The residual effects of problems tagged as I can do otherwise
is another. The level of semiosic commitment to unproductive sense making
distinguished by Oops!, Where am I?, and I cant do it this wayis yet
another example.
One of the major advantages of using input from other theories at the interpre-
tation stage is that the evaluators semiosis is boosted by the complementarity and
articulation among various HCI theories, thus broadening the scope of her analy-
sis. This can only be benecial for the evaluation, especially if more than one theory
or approach is related to the semiotic one. For example, in addition to pairing com-
municability breakdown categories with the cognitive issues raised in cognitive
walkthroughs, the evaluator gains insights from pairing them with some popular
usability issues examined in heuristic evaluation (Nielsen 2001) (see table 4.3).
This pairing is different from the previous one because, as its name says, heuris-
tics are principles derived from practical experience and observation and are not
typically committed to ontologies and categorizations that are used in theoretical
accounts of the observed phenomena. Therefore, unlike the pairings supported by
theoretical associations that can be established between cognitive and semiotic phe-
nomena in table 4.2, the pairings in table 4.3 are both less obvious and weaker than
the previous. At times, the tags associated with one heuristic principle may seem
best t for another association. For instance, the Thanks, but no, thanks tag sug-
gests a stronger pairing with the User control and freedom principle than with
the Match between system and the real world one. However, a high frequency of
occurrence of Thanks, but no, thanks suggests that what users do or prefer in
the real world is not what has been encoded in the system. This tag may also be
Communicability and the Designers Deputy Discourse 143
144 Chapter 4
Table 4.3
Pairing potential HCI problems dealt with in heuristic evaluation and communicability
evaluation.
Heuristic evaluation issues Communicability breakdown categories
Visibility of system status Looks ne to me.
What happened?
What now?
Where am I?
Match between system and the real world I give up.
Looks ne to me.
Thanks, but no, thanks.
I can do otherwise.
User control and freedom Thanks, but no, thanks.
I can do otherwise.
Consistency and standards Where is it?
Where am I?
Oops!
Error recognition and recovery Looks ne to me.
I can do otherwise.
Where am I?
Oops!
I cant do it this way.
Help!
Why doesnt it?
Error prevention Looks ne to me.
I can do otherwise.
What now?
Where am I?
Oops!
I cant do it this way.
Help!
Why doesnt it?
Recognition rather than recall Where is it?
What now?
Where am I?
Oops!
Whats this?
Help!
Flexibility and efciency of use Thanks, but no, thanks.
I can do otherwise.
Aesthetic and minimalist design (none)
paired with the Flexibility and efciency of use principle, denoting that the range
of illocutions that can achieve the same desired perlocution is wide.
These many-to-many correspondences between heuristic principles and commu-
nicative breakdowns that can be brought about when they are not observed in
design might be interpreted as an indication that this pairing is not productive at
the interpretation stage of communicability evaluation. If the occurrence of tags
suggests one of many breaches of a heuristic principle, and vice versa, what is the
pairing actually telling the evaluator?
At least two lines of thought lead to interesting answers to this question. One is
that the correspondence may in itself show some clusterings of heuristic principles.
For example, Error prevention obviously goes with Error recognition and recov-
ery, and maybe less obviously with Help and Documentation. The occurrence
of one or more tags that can be associated with any of these principles will proba-
bly show how these principles can or should be articulated with each other to resolve
a communicative problem. Conversely, it will probably show communicative prob-
lems that stem from inappropriate articulation of such heuristic principles in design.
The same applies to other related principles such as User control and freedom
and Flexibility and efciency of use, Consistency and standards and Recog-
nition rather than recall, and so on. The other line of thought that shows the benet
of pairing heuristic principles to communicative breakdown categories is to trace
the various dimensions of design that contribute to good designer-to-user meta-
communication. For instance, a Looks ne to me tag signals a single basic cate-
gory of communicative breakdowna complete communicative failure of which the
user is not aware. Pairing this tag to such heuristic design principles as Help and
documentation, Error prevention, Error recognition, Match between system
Communicability and the Designers Deputy Discourse 145
Table 4.3
(continued)
Heuristic evaluation issues Communicability breakdown categories
Help and documentation I give up.
Looks ne to me.
Thanks, but no, thanks.
I can do otherwise.
Whats this?
Help!
Why doesnt it?
and the real world, and Visibility of system status calls the designers attention
to the factors that contribute to efcient communication. Therefore, his awareness
of the importance of using familiar signication systems, of enabling implicit and
explicit metacommunication dialogs, and so on may be raised. This pairing also
enriches the answer to a question posed frequently by designers after their products
are evaluated: And so what? Sometimes usability testing and evaluation provides
designers with a list of problems found in their design, classied according to one
or another taxonomy preferred by the evaluator. But it may not be clear how to
prevent the recurrence of the problem. Designers may have done their best to follow
a particular principle, yet the results show that at least in the case of a particular
piece of design, their intent did not come through. The association of some of these
principles with parts of a theory of HCI will allow them to step ahead of ad hoc
diagnostics and enable them to think more broadly about the very process of design.
Table 4.3 also shows that some elements of one approach may not be dealt with
at all in communicability evaluation. Such is the case of aesthetic and minimalist
design. Although Mullet and Sano (1995) provide us with the means to explore
the associations between aesthetic and minimalist design and communicability, the
focus on the core ontology of semiotic engineering becomes blurred along these lines
and the analysis becomes less productive. Again, this shows the complementarity of
approaches and helps the adopter of semiotic engineering see the boundaries of this
theory. Other types of items that are not easily associated with the categories
accounted for in communicability evaluation can be found among ergonomic crite-
ria, compiled by Bastien, Scapin, and Leulier (1999) in the form of guidance, work-
load, explicit control, adaptability, error management, consistency, signicance of
codes, and compatibility. Although signicance of codes and guidance might
suggest that the pairing would be promising, the ergonomic perspective is not so
directly related to the semiotic one (except through the associations already dis-
cussed in relation to the cognitive walkthrough and heuristic evaluation). For
instance, conciseness, minimal actions, decisions about location and
format of signs, and information density may all have a semiotic stance, but
semiotic engineering does not explore these dimensions in greater depth. Commu-
nicability evaluation can be stretched to cover these issues, but the results may not
be particularly illuminating or better than using other more direct approaches.
4.3.3 Generating the Semiotic Prole of Designer-to-User Metacommunication
The semiotic proling stage nalizes the whole process of communicability evalua-
tion. In it an in-depth characterization of the designer-to-user metacommunication
146 Chapter 4
is achieved. The designers message can be spelled out by the evaluator, who will be
able to assume the rst person in discourse and say the following:
Who do I think are (or will be) the users of the product of my design?
The answer to this question mainly reveals the cultural, psychological, and techni-
cal characteristics of the listener of the designers metacommunication message. The
full range of communicative breakdowns presented so far in this chapter will detect
matches and mismatches between the designers intended listener and the actual
listeners that represent the constituency of users that the design product will
reach.
What have I learned about these users wants and needs?
The answer to this question covers the range of perlocutions achieved by the illo-
cutions that the signication systems used in the design support. Complete com-
municative failures will distinctively indicate severe breakdowns that must be
corrected for HCI to fulll its ultimate goal. Moreover, the distinctions among dif-
ferent goal-related levels of problems will help spot ne-grained mismatches between
the designers illocution (i.e., what he intended to say with the design) and the per-
locution it achieves (i.e., what users get from it and do with it).
Which do I think are these users preferences with respect to their wants and
needs, and why?
The answer to this question is mainly the designers justication for the signica-
tion systems he has used and for the rhetoric that can or must be used for valid illo-
cutions that these systems support in the process of communication. A signicant
incidence of I can do otherwise and Thanks, but no, thanks tags in the evalu-
ation will probably mean that the designers understanding and rational justica-
tion for the decisions he made are not consistent with the real world.
What system have I therefore designed for these users, and how can or should
they use it?
The answer to this question tells how well the expression and content of the
designers metacommunicative illocution is being transmitted to the user. The
expression will be directly connected to the actual signs that are used in the inter-
face and that emerge during interaction. As a result, problems with the users per-
ceiving and interpreting such signs (such as those indicated by, e.g., What
happened?, Whats this?, Why doesnt it?, and Help!) will direct the
designers attention to a more efcient use of the users cultural background. Like-
Communicability and the Designers Deputy Discourse 147
wise, the content of the metacommunicative illocution will be directly connected to
the users ability to derive productive semiosis from the expressions in the signi-
cation system. The I can do otherwise tag tells the designer that users are not
getting his message, even if they are able to achieve short-term goals with relatively
little trouble. It calls the designers attention to the therefore that is deliberately
present in the formulation of this semiotic proling question: What is the cause of
the users missing the content of the message? If the answers to the previous ques-
tions have already indicated sources of communicative problems, these sources are
very likely the cause of the problem here as well. However, the preceding questions
may have indicated relatively minor problems, in which case problems may stem
from the complexity of communicating with the system itself. For instance, taggings
dominated by Oops! and I cant do it this way are likely to mean that there
are illocutionary obstacles in the design that the user is not able to overcome so
easily. Such tags dont mean that the users background, their wants and needs, or
their preferences havent been taken into account. They only mean that the engi-
neered sign system is difcult to use.
What is my design vision?
The answer to this question indicates how well the design rationale has been under-
stood (and accepted) by the user. The occurrence of Thanks, but no, thanks is
likely to mean that it was understood, but with some restrictions. In other words,
the perlocution achieved by the designers message does not fully coincide with his
intent. This fth question asked at the beginning of the semiotic proling stage is
particularly discriminative if there havent been signicant problems spotted in the
answers to the previous questions. Specically, it may help the designers evaluate
the return on their investment in design. For example, if the application enables and
highlights the use of certain conversational paths whose design and/or implemen-
tation have consumed many resources, the users refusal to follow them is a very
bad sign. Another interesting aspect of this question is that it can handle some
branding and marketing messages, which are part of the design but have little or
nothing to do with usability and other popular -abilities handled by alternative
evaluation methods.
The semiotic proling also makes an extensive use of the concepts presented in
chapter 2. Three of them stand out as particularly productive at this stage: the com-
municative functions derived from Jakobsons model for communication; the prag-
matic principles proposed by Grice; and the parameters of Ecos TSP.
148 Chapter 4
Jakobsons model, which has been the source of fundamental concepts in the
semiotic engineering ontology, will help the evaluator walk through the commu-
nicative design space and inspect the rhetoric of the designers deputys discourse.
By contrasting this rhetoric with her ndings from the tagging and the interpreta-
tion stages of the communicability evaluation method, the evaluator will assess the
efciency of metacommunication. She may use the Gricean maxims and Ecos sign
production parameters to explain inefciencies and to suggest potential improve-
ments. For example, she may nd that the designers deputys discourse is lacking
in metalinguistic functions. That is, the designers metacommunication does not
make an efcient use of communication about the designers message itself. Online
help functions, information, and explanations during error prevention and recov-
eryall of these may have been poorly explored by the designer. The Gricean
maxims will help her indicate the appropriate locus and quantum of metalinguistic
functions that the application can use to increase communicability. Ecos parame-
ters will help her express the metalinguistic functions in more efcient ways.
Moreover, bringing together metaphors and metonymies, on the one hand, and
abductive reasoning and semiosis, on the other, the evaluator may even be able to
identify the design cues mentioned by Brown and Duguid (1992) that are enhanc-
ing or hindering the communicability of a particular application. For example,
abductive reasoning and semiosis are based on the universe of hypothesis motivated
by the signs present in a particular situation. So, in the Eudora scenario described
at the beginning of this chapter, the frequency of successful menu options as a way
to communicate his intent reinforces the users hypothesis that the appropriate illo-
cution to switch signatures is a menu option as well. The problem of encoding illo-
cution as a list selection is that pull-down lists are far less salient in the interface
than menus (see the screen dumps in the scenario and check that there are only three
pull-down lists in the main window, against more than three times as many menus).
So it is only reasonable that the user will engage in a long series of breakdowns
stemming from menu option explorations. Some important metaphorical and
metonymical expressions have been cued by the user along the way. For instance,
his suspicion that the option Alternate might refer to creating and editing the alter-
nate signature, but also to switching to it, points to metonymical interpretation
of the sign as standing for anything related to (contiguous to) the literal meaning
of alternate signature. Another metonymical axis has emerged in interaction
when the user has tried to nd illocution in the Message menu. Again, the sign
message might reasonably mean anything related to messages, especially in the
Communicability and the Designers Deputy Discourse 149
presence of a change option embedded in this menu. Finally, a powerful
metaphorical expression has been used by Joe when he copies and pastes the textual
content of the alternate signature and nds he has solved the problem. It denotes
his intuitive understanding of what the signature procedure does (internally) and
foreshadows design enhancements achieved in subsequent releases of Eudora by
Qualcomm.
All of these elements may be explored in the semiotic proling and produce an
in-depth characterization of the applications communicability along with sugges-
tions for improving the designers deputy discourse. Another valuable procedure in
communicability evaluation is to contrast taggings made by the evaluator, the
designer himself, and occasionally the user. The designer has a privileged perspec-
tive on metacommunication, since he is the one who has built the designers deputys
discourse and, consequently, the one who knows exactly what was meant by each
and every illocution. The user, however, has some interesting epistemological limi-
tations for the tagging. For example, the user cannot be expected to tag interaction
with Looks ne to me unless he suddenly becomes aware of the fact that what
he thought was an acceptable end state for his activity is in fact not one. The same
applies to I can do otherwise tags, unless the user senses that there is a better
way to tell the system what to do, although he is not capable of nding it. The
authors experience with teaching communicability evaluation in graduate and
undergraduate courses is that tagging interaction is an enjoyable learning experi-
ence for future HCI designers and researchers. The suggestion is that the utterances
are easy to learn, remember, and use effectively. For this reason, as will be seen
in section 4.4, they have been used as part of an expressive code that users can
resort to in order to engage in explicit metacommunication with the designers
deputy.
4.4 Improving the Designers Deputy Discourse
Communication is virtually impossible when interlocutors lack the ability to negoti-
ate meanings. And one of the keys to negotiating meanings is the interlocutors ability
to explain what they mean by what they are saying. When transposed to HCI, this
requirement sheds light on some fundamental issues. First, natural languages and
computer languages are structurally distinct when it comes to supporting what
Jakobson calls the metalinguistic function in communication (see chapter 2).
Whereas natural language allows us, humans, to produce as many levels of metadis-
course as we want to, computer languages are bounded by self-description limita-
150 Chapter 4
tions imposed on every formal language (Hopcropf and Ullman 1979). In brief, com-
putable languages cannot be used to specify the full range of their own semantics,
which means that a system can only provide complete explanations about what it
means if another system (that can interpret and produce the metalanguage of the
original system) can tell it so.
3
Therefore, unlike humans, who can always explain
what they mean by what they say, technically speaking, computer systems often have
to ask other computer systems for an answer to this question. This constitutes what
Winograd and Flores (1986) refer to as systems that cascade levels of representa-
tion one on top of another to great depth (87). Even if what we call the system is
often a collection of numerous programs, each one specializing in interpreting and
generating a particular level of representations (or, strictly speaking, a particular sig-
nication system), there must always be some other system to explain what the
topmost representation-processing program means by most of what it says.
It follows from the preceding logic that the explanatory power of the designers
deputys discourse, and consequently its ability to negotiate meanings, is limited by
formal properties of computable languages. Even if it is very powerful by some
practical criteria, or powerful enough within the range of acceptable perform-
ance, it is never comparable to what humans can do with and because of language.
So the encoding of the designers meaning negotiation ability (which is performed
by his deputy) is a key issue for successful communication with the user. This
involves two different processes: that of detecting when the users illocution is
incomprehensible and must therefore be claried, and that of explaining what the
designers deputys own illocution means. Both tasks are particularly difcult and
complex for computer programs, as efforts in natural language understanding have
shown (Moore 1995). The semiotic engineering alternative in order to maximize
the communicability of the designers deputys discourse is to enrich the form and
content of online help.
4.4.1 The Problem with Online Help
In the 1980s, Carroll and Rosson (1987) spotted the paradox of the active user.
Although knowing more about the application one is using is certainly a factor that
promotes much better HCI and much higher productivity, users dont read manuals,
dont use online help systems, and prefer to jump right into the doing instead of
rst learning what can be done. Evidence from numerous empirical studies about
users behavior led to minimalisman action-centered approach to instruction
where the context of real activities is the key to learning (Carroll 1998). Minimal-
ist techniques used in online help systems, such as layering (Farkas 1998), seek to
Communicability and the Designers Deputy Discourse 151
provide just-in-time and just-as-much information for users, even at the risk of not
giving enough information for the user to complete the task (a case of I give up),
or consuming a lot more of the users resources than is actually needed (a case of
I can do otherwise), or facilitating the elaboration of equivocal mental models
that will cause problems in interaction later on (a case of I can do otherwise or
Looks ne to me). These three risks have been spelled by Farkas (1998), for whom
the degree of the risk depends on how radically information is cut and just what
is cut and how (249).
What is cut and how is a decision that we humans make all the time whenever
others ask us for information. We intuitively follow the Gricean maxims for coop-
erative conversation (see chapter 2) and make extremely ne-grained contextual
judgments about the quantity, the quality, the relevance, and the form of informa-
tion that must be provided. Such judgments lie outside the scope of even the most
sophisticated expert systems, and in part this is why online help systems have a bad
reputation for not helping you much unless you know the kind of help you need
and know exactly how to ask for it. Most of the information they contain is related
to the goals, plans, and operations involved in bridging Normans execution gulf
(1986). Its mainly about tasks, and not so much about the design decisions that
support and explain the way in which they can or must be carried out in order to
make computational sense. Knowing that such decisions exist and the extent to
which they determine the quality of a computer artifact is the essence of true
computer literacy.
Help systems are not typically designed to enhance computer literacy. Be it because
computer literacy is an elusive concept, and/or because users are known to be both-
ered by intrusive instructions on their way to getting things done, the fact is that help
systems concentrate on telling them just how to get things done, preferably in a min-
imalist way. The problem is that the users semiosis is permanent and unbounded. So
users will wonder and guess many thingsright and wrong. Many of their guesses
will have to do with the whys and for whats that help systems dont usually
account for. Communicability evaluation is an efcient method for studying interac-
tive patterns that are related to the users wondering and guessing. Therefore, com-
municability utterances can be used to provide access to minimalist information
about other dimensions of the artifact, especially design rationale. This kind of infor-
mation may improve meaning-negotiation processes between the user and the
designers deputy at interaction time, because it widens the scope of signs that can
play a positive role in motivating productive semiosic paths. This is the semiotic engi-
152 Chapter 4
neering alternative to enhancing the designers deputys explanatory discourse
without having to resort to AI techniques. By expanding the scope of information
content to topics that are not exclusively related to goals, plans, and operations, this
alternative takes its cue from ethnography (Suchman 1987) and activity theory
(Bdker 1989). A larger part of the users opportunistic semiosis may be positively
inuenced by traces of the designers rationale appearing during interaction.
4.4.2 The Semiotic Engineering of Online Help Systems
The kinds of information that are necessary for enriching the designers deputys
discourse can be derived from various design models, such as user models, domain
models, task models, interface models, and others. Therefore, the semiotic engi-
neering of HCI is intrinsically related to model-based design (Patern 2000).
Silveira (2002) proposes that six design models be used in the construction of
explicit metacommunication messages: a domain model, a user model, an applica-
tion model, a task model, an interaction model, and an interface model. Different
modeling techniques may provide input models for each of Silveiras models.
Puertas user-task, domain, presentation, dialog, design, and user models, for
instance, are an alternative (Puerta 1996). Nevertheless, special attention should be
given to differences in perspectives that guide the modeling activity itself. For
example, there is a fundamental difference between Silveiras and Puertas
approaches, which requires that Puertas models be enriched to meet Silveiras
requirements. Puertas primary goal in working with HCI models is to solve
mapping problems and enable the design of a wide variety of user interfaces
previously unattainable using model-based technologies (Puerta and Eisenstein
1999, 177). In other words, the focus is on supporting developers with automated
design tools and environments that will enhance the consistency of interface design.
Silveiras primary goal, however, is to support the elaboration of online help systems
(Silveira, de Souza, and Barbosa 2001a, 2001b). The purpose of the information in
Silveiras and Puertas models is very different. Consequently, the perspective of
information content is also very different. Puertas user-task model can be instanti-
ated with a GOMS model (Card, Moran, and Newell 1983; Puerta 1996), which
contains the hierarchical specication of plans and operations that are needed for
the accomplishment of certain goals. In Silveiras approach, task models must be
enriched with certain types of information that are usually not included in generic
hierarchical models of tasks and activities. Semiotic engineering task models require
the following top-level categories of information:
Communicability and the Designers Deputy Discourse 153

the hierarchical structure of tasks, subtasks, and operations required for the
achievement of top-level goals, with their respective

preconditions (if any);

relative ordering (or absence of ordering);

possibilities of iteration (none or one, one or many, any number, some specic
number);

alternative patterns (which mean that two or more of them can be used to achieve
the same effect);

associated signs from the signication system (independent of their being explic-
itly or implicitly referred to during interaction); and

error prevention or error handling mechanisms.

the type of task, subtask, or operation with respect to its content, form, or
illocution:

standard structured and nonstructured types;

stereotypes that recur in different parts of the hierarchy with specic parameter
variations; and

ubiquitous operations, tasks, or subtasks of the hierarchy, which can be performed

at any moment (e.g., abandoning the activity, saving data, and so on).
The most important property of such extended task models in semiotic engi-
neering is that they provide an index to interaction models. They point at the
specic types of conversations where the users and the designers deputys
illocutions bring about the perlocution specied in the task model. This is certainly
an instance of Puertas mappings, except that its purpose is fundamentally com-
mitted to communicative dimensions of speech acts, such as illocution and per-
locution. Notice that Puertas dialog models express the designers deputys
illocutions as a class of relation between commands issued by the user (in our terms,
the users illocution) and a certain state of affairs that follow it (the corresponding
perlocution). An explicit emphasis on distinguishing between the perlocution of the
users illocutionary acts and the designers deputys illocution following the users
perlocution is precisely what will make a fundamental difference in terms of
metacommunication.
For example, in the Eudora scenario that dominates this chapter, the perlocution
of Joes having commanded (in Puertas terms) the creation of an alternate sig-
154 Chapter 4
nature is that the created signature can now be part of other illocutions (like Switch
to the alternate signature or Edit the alternate signature). And this is precisely
the users intent. The problem is that the designers deputys subsequent illocutions
are all problematic. For instance, they do not clearly say that the alternate signa-
ture is there. Neither does it clearly say how to formulate an illocution meant to
switch signatures when a message is composed. As communicability evaluation
shows, perlocution and illocution breakdowns belong to different categories of
problems and must therefore be treated differently if we really want to help the
users nd their way in interaction. This is why all semiotic engineering models must
privilege categories of information that will directly affect communication between
users and designers deputies.
The kinds of interaction model used by Silveira require that the following types
of information be identiable (Barbosa and de Paula 2003; de Paula 2003):

interaction scenes, which all have

a topic, which is indexed to a goal, task, or subtask in the task model;

a conversation, which is specied by a sequence of alternate illocutions performed

by the designers deputy and the user;

an associated set of system processes that determine the global perlocutions

achieved in the scene; and

an associated set of transitions that determine the conversational paths that

precede and succeed global perlocutions achieved in the scene.

illocutionary components, which all have

an identication of the interlocutor;

a generic natural language representation of an illocutions content;

a generic representation of the illocutions context (expressed as preconditions);

and

a generic representation of entailed perlocution (expressed as postconditions).

The formalisms that can be used to specify semiotic engineering task and inter-
action models in detail are not important, provided that they support the expres-
sion of the types of information that these models must contain. In gure 4.4 a
mainly visual characterization of the task and interaction models for a typical login
activity shows how the elements in them are related. The alternation of illocutions
by the user (u:) and the designers deputy (dd:) in the interaction model denes
the illocutions and conversational patterns (at an abstract level) that achieve the
Communicability and the Designers Deputy Discourse 155
perlocution associated with the Request registration goal. Notice that conversa-
tions about errors, as well as conversations about abandoning the current goal, are
explicitly dened in the interaction model based on specications from the task
model. The elements of these models will provide crucial information for resolving
communicative breakdowns expressed by communicability utterances. For instance,
What happened? requires information about the (form, content, intent of the)
last illocution performed by the designers deputy. Where is it? demands infor-
mation about the range of illocutions that the user may perform given the designers
deputys previous illocution and the perlocution achieved through it. I cant do it
this way requires information about the scenes associated to the goals that users
can achieve through the application.
156 Chapter 4
Figure 4.4
Types of information required in task and interaction models.
Silveira proposes that help systems be prepared to use model information to
address the following questions:

Who are the applications users?

What do they usually do in their domain of activity?

What kind of intervention is the applications technology expected to achieve in

the users domain?

Which domain objects are present in the application?

What activities and tasks can be performed?

Why are activities and tasks necessary? Whom do they affect? Whom or what do
they depend on?

What signs compose the interface signication system?

How does interaction proceed?

When and why is one kind of interaction better than another when they achieve
the same effect?
She extracts the answers to these questions from the six models she uses to
compose the designers deputys metacommunicative discourse (Silveira 2002). From
the domain model, she derives domain signs that are implicit and explicit in the
interface signication system. Each sign has an expression (or name), content (or
description), and associated illocutions (or utility). From the application model,
she derives the application signs that are implicit and explicit in the interface signi-
cation system. Like domain signs, each application sign also has an expression (or
name), content (or description), and associated illocutions (or utility). From
the user model, Silveira extracts users roles and proles. The proles contain the
users identication and characterization (through relevant properties). User roles
associate certain groups of users (or proles) with certain tasks. From task and inter-
action models Silveira derives operational and tactical information. An important
aspect of this information is the ability to distinguish between domain-related signs
and application-related signs. While the former can be expected to be reasonably
known by the users, and demand briefer explanations, if any at all, the latter result
from a technological intervention in the domain, which is likely to require much
more elaborate explanations and illustrations. Finally, from the interface model Sil-
veira extracts sign types and tokens, as well as the kinds of illocutions in which they
may appear.
Although the full spectrum of information in the designers deputys explanatory
discourse might suggest that online help systems proposed by semiotic engineering
Communicability and the Designers Deputy Discourse 157
are too complex (especially for the user), three strategies are used to simplify meta-
communication and make it more attractive and useful. First, as already mentioned,
the designers deputys discourse follows the minimalist approach and layering tech-
niques proposed by Farkas (1998). This promotes more cooperative communica-
tion with the user, who ideally gets just-in-time and just-as-much information as he
needs to proceed with semiosis. Second, metacommunication pervades all interac-
tion and is not restricted to what is traditionally considered the online help func-
tion. It is presented in traditional form, but also through localized minimalist
conversations. Moreover, the content of error messages and direct instructions
is carefully designed to motivate productive semiosis, given their context of occur-
rence during interaction. Finally, the third strategy that facilitates explicit meta-
communication between the user and the designers deputy is the use of
communicability utterances and typical phrases that express the need for help (Sellen
and Nicol 1990). Table 4.4 illustrates utterances appearing in Silveiras localized
158 Chapter 4
Table 4.4
Sources of metacommunication contents.
Element of localized interaction Communicability utterances and help expressions
Domain and application signs Whats this?
Whats this for?
Where is it?
Task structure (conversational intent Whats this?
and perlocutions) Whats this for?
How do I do this?
Oops!
What now?
What happened?
Who is affected by this?
What does this depend on?
Who can do this?
Where was I?
Sequence of actions (conversations) Why do I have to do this?
Is there another way to do this?
Actions (speech acts) How do I do this?
Oops!
What now?
What happened?
Where was I?
minimalist conversations and the corresponding source of metacommunication
contents.
The effect of Silveiras model is illustrated in gure 4.5, where local meta-
communication enabled during a request for registration to an online service is
exemplied. The example is taken from an existing Bulletin Board application (de
Souza, Prates, and Barbosa 2003) developed by the Semiotic Engineering Research
Group for a Brazilian organization of social volunteersAssociao Sade-Criana
Renascer (ASCR). The textual content, all in Portuguese in the original application,
Communicability and the Designers Deputy Discourse 159
Figure 4.5
A screen shot from an implemented online help system that uses Silveiras model.
has been translated into English by the author. The smaller browser Bulletin Board
Help windows are opened when the users click on the blue question marks posi-
tioned next to certain interface elements, or when they click on a link that appears
in the metacommunication message. The screen in gure 4.5 results from the users
rst clicking on the question mark next to Request Registration, then clicking on
the Bulletin Board link. Notice how the communicability utterances and help expres-
sions are used to provide minimalist insights about the application. It is the HCI
designers task to dene the kind of metacommunication associated with each
element of the interface, and to select the utterances and expressions that will
provide access to the designers deputys explanatory discourse. He must also iden-
tify the points of recurrent metacommunication, expressed as navigation links in
the textual content of the designers deputys message. The complete method for
building online help systems is presented in previous work by Silveira, de Souza,
and Barbosa (2003).
4.5 Epistemic Tools for HCI Evaluation and Explanatory Discourse Design
This chapter introduced various epistemic tools that HCI designers can wield to
reect on the nature and the possibilities of metacommunication that can be
achieved through computer artifacts. It started by extending the existing notion of
communicability in order to account for all signs belonging to the applications sig-
nication system. It then proposed a method for evaluating the communicability of
interactive computer artifacts, based on specic categories of communicative break-
downs. Such categories result from discriminatory principles centered on the two
major dimensions of speech acts: illocution and perlocution.
The communicability utterances in themselves are a kind of epistemic tool in that
they trigger the designers semiosis in the direction of communicative breakdowns
that must be prevented or circumvented in HCI. A more systematic use of this tool
produces another set of tools presented in this chapter: the models and techniques
that contribute to building the designers deputys discourse conveyed through online
help. One is the use of communicability utterances and help expressions as a means
to express the users need for meaning negotiation, through explanations and clar-
ications. Another is the range of information that can be extracted from domain
models, user models, application models, task models, interaction models, and inter-
face models to compose the content of metacommunication messages sent through
traditional online help, localized minimalist conversations about the artifact, error
160 Chapter 4
messages, and instructions. In particular the additional information included in hier-
archical task models and the communication-oriented interaction models themselves
widen the scope of metacommunication without requiring specic AI techniques
used in explanatory dialogues (Moore 1995). Designers may simply use templates
to generate the appropriate responses to the users questions about the application.
These templates can be generated based on a systematic correspondence between
communicability utterances and help expressions, on the one hand, and informa-
tion extracted from models, on the other.
Communicability and the Designers Deputy Discourse 161
5
The Semiotic Engineering of Customizable and
Extensible Applications
In previous chapters we saw that the designers deputy is a powerful abstraction in
the achievement of successful metacommunication. It will convey to users the orig-
inal design vision and have a range of conversations about it in order to clarify
aspects of this vision and reinforce the users semiosis toward interpretations that
bring them closer to understanding the designers intent. But we also saw that
whereas humans interpret signs as part of an unlimited semiosic process, practical
computer programs can only interpret them as part of algorithmic symbol pro-
cessing.
4
This semiotic gulf between users and systems is of a different nature than
the execution and evaluation gulfs characterized in cognitive engineering (Norman
1986). The semiotic gulf can only be bridged, although partially, if computer systems
begin to accommodate expansions in the semantics of encoded signication systems
that users are exposed to while interacting with computer applications. Only then
can users experience something that somehow resembles human semiosis. Meaning
expansions are regularly required for the success of human communication, both
from a message senders and a message receivers point of view. Take the following
dialog as an example:
Speaker 1: I see that some system components should not be accessible for
changes. Its as if they were impermeable to customization and extension.
Speaker 2: Maybe. But what sort of coating would you use to help users
identify permeable from impermeable components? It should not be a matter of
trial and error, I guess. Nor of hiding these components, otherwise they would
be invisible and not impermeable.
Speaker 1 has expressed himself using a comparison (or simile). Speaker 2 has
immediately expanded the semiotic space of the conversation and talked about
coating (meaning an impermeable protection), adopting a metaphorical expres-
sion that leads naturally to other relevant aspects of the discussiondistinguishing
between impermeable and invisible things. Speaker 2 is helping Speaker 1 elucidate
his own meanings (see Jakobsons metalinguistic function in chapter 2) by using a
rich form of expression that associates unusual (if not strictly incorrect) properties
to systems components.
Meaning expansions introduced by users have been the object of investigation of
end-user programming (EUP) and more recently end-user development (EUD).
Meaning adaptations of various sorts, not necessarily expanding the semantic base
of encoded signication systems, have been the object of studies about customiz-
able (and at times personalizable) interfaces. In this chapter I discuss the importance
of EUP and customization from a semiotic engineering perspective.
Viewing interactive software as metacommunication artifacts highlights the fact
that designers want to achieve a perlocutionary act when they send the metacom-
munication message to users. The message is encoded and communicated in an arti-
cial signication system that is continuously processed by symbol-interpreting and
symbol-generating computations triggered by interactive events. The range of
symbol (or sign) combinations that can be interpreted and generated is technically
innite once the system is encoded as a true language. For instance, rules about the
property name (that users can assign to les they create through various appli-
cations) typically characterize an innite number of alphanumeric chains of char-
acters. In other words, the set of all names can only be formally specied, but
not exhaustively listed. This simple principle extends to much more elaborate and
structured grammatical combinations that are systematically interpreted as system
component properties, states, behavior, and so on.
When designers construct a language to convey their message to users, as well as
subsequent user messages to the designers deputy and vice versa, they are building
semiotic innities whose boundaries cannot be exhaustively known. Figure 5.1 iden-
ties four different regions in the territory dened by linguistic specications of sig-
nications codes. The central region in it is that corresponding to the designers
targeted perlocutions. Simple and compound signs and symbols in it refer to what
the designer consciously meant to tell users (and consequently to get users to do
with the system). Traditional online help usually depicts the range of utterances that
belong to this region. The next surrounding region, in the ripple shape presented in
gure 5.1, is that of simple and compound signs and symbols that are in accordance
with the encoded system (i.e., they are grammatically correct) but that are not appro-
priate or useful in view of the designers pragmatic goals. Utterances of this type
164 Chapter 5
typically result in warnings, error-preventing dialogs, error-handling instructions,
and the like. Just like the previous type (targeted utterances), these are expected
utterances, even if they do not lead to users or designers full communicative sat-
isfaction. By predicting what users must and may say to express what they want to
do, designers will provide the necessary mechanisms to effect what can be done and
to prevent and explain mistakes when things cannot be done. However, there are
other utterances that fall within the range of well-formed symbol or sign combina-
tions, but that designers havent really gured out at design time. These are the ones
that cause the most disorienting interactive failures in HCI, stemming from mes-
sages like A fatal error has occurred or Index out of bound. They are related
to interactive contexts that the designer has not thought of, and has thus failed to
prevent and explain appropriately, thereby not giving the user a chance to under-
stand what happened and what must or can be done next. Finally, although there
are technically innite utterances falling within the encoded signication system,
there are also innite utterances falling without. The absence of a semantic type like
converted formats in a text editor, for instance, may never enable expressions
such as Save as HTML. This combination of symbols (from a computers per-
spective) or signs (from a human perspective) does not belong to the interface
language that can and must be used by users and system. But this semantic type
(converted formats) and one of its instances, such as HTML, is not a surpris-
ing sign in the semiosic processes of any contemporary user of text editors.
Figure 5.1 provides a background for understanding the complexity of com-
municating ideas, visions, principles, rules, and constraints through messages that
can interpret and generate other messages. The fact that such communication is
achieved through the use of an articial code, or language, specically designed for a
The Semiotic Engineering of Customizable and Extensible Applications 165
Figure 5.1
Different signication territories associated to interface signs.
particular metacommunication act, turns this language into a unique and arbitrary
(in the Saussurean sense explained in chapter 2) signication system that users may
wish or need to adapt to their evolving semiosic processes. This is the main motiva-
tion for developing customization and extension techniques that allow users to change
the forms and/or meanings in the originally designed system. If all forms and mean-
ings could be changed at leisure by users, however, this would allow users eventually
to destroy the original designers message and build another one of their own. This
would be the equivalent of reprogramming the application through manipulating the
signication system embedded in it. Reprogramming would not only be a difcult
(and probably unattainable) goal for the vast majority of users, but it wouldnt actu-
ally make sense in the communicative perspective adopted by semiotic engineering. It
would only make sense in a full-edged programming and development environment
where the designers goal is that users create their own signication systems and mes-
sages encoded therein. So customizable and extensible applications must have signs
and symbols whose form and/or meaning cannot be changed by users. They must be,
so to speak, impermeable to changes in the signication system.
Impermeable signs and symbols constitute the identity of customizable and exten-
sible applications. They are a permanent reference for symbol processing and semi-
osis and have the power to preserve the gist of the designers metacommunication
message. They must therefore be carefully designed and used in all communicative
processes enabled by the application, including those that will introduce changes in
communication itself. It is thus helpful to analyze the semiotic dimensions involved
in changing signication systems from a systems and from a users perspective.
5.1 Semiotic Aspects in Changing Signication Systems: The Computer
Perspective
From a computers perspective, a signication system is a language, or a system of
related languages. And there are essentially three semiotic dimensions that computer
systems can handle in processing languages: lexical, syntactic, and semantic. The
lexical dimension accounts for the vocabulary that can be used to form meaningful
sentences of the language. The syntactic dimension accounts for the kinds of com-
binations in which vocabulary items can be arranged and the structures, or higher-
order symbols, that are created with such combinations. Finally the semantic
dimension accounts for two fundamental operations in linguistic computations: It
establishes the appropriate mappings between vocabulary items or higher-order
166 Chapter 5
symbols and representations or actions corresponding to what they mean; and it
effects the operations associated with the computed meanings. The semantics of
computer languages is often described in algebraic and procedural formalisms, or
in natural language text (which handles both). The concepts introduced in chapter
2 help us see that computer languages merge the semantic and pragmatic dimen-
sions of natural languages into the same linguistic component, understandably
associated with computer language interpreters (i.e., sets of functions that can
understand and effect what well-formed linguistic expressions mean).
A computer language can be altered in various ways. There may be lexical changes
(followed or not followed by syntactic and/or semantic changes), syntactic changes
(followed or not followed by lexical and/or semantic changes), and semantic
changes (followed or not followed by lexical and/or syntactic changes). Figure 5.2
presents all the possibilities for manipulating, independently or jointly, the three
dimensions specied in computer languages.
In the context of customizing and extending applications, it is important to notice
the threshold between changes of types I, II and III, on the one hand, and those of
types IV, V, VI and VII, on the other hand. The rst three are meaning-preserving
changes, whereas the others introduce meaning modications in the encoded
language. This helps us establish a formal distinction between customization
(meaning-preserving alterations) and extension (meaning-changing alterations),
which we will adopt throughout this chapter.
By preserving syntax and semantics intact, type I changes amount to renaming
vocabulary items or introducing synonyms. This kind of customization can be done
in many applications, although they usually have a number of items that cannot be
renamed, for example. They are impermeable signs that constitute the applications
The Semiotic Engineering of Customizable and Extensible Applications 167
Figure 5.2
Possibilities for changing computer language dimensions.
identity (be it for HCI design or software implementation reasons, or both). The
interest of associating impermeability to identity is to include the design of the appli-
cations identity as an explicit and fundamental step in the development of cus-
tomizable and extensible applications.
By preserving vocabulary and semantics intact, type II changes essentially amount
to changing the combinations prescribed by grammatical rules. It is not very useful
to introduce new grammatical types if the vocabulary and semantic rules to inter-
pret them remain the same. It also not very useful to introduce new higher-order
symbols in the grammar if they do not advance any other interpretive aspect of the
language. Thus, type II changes essentially enable the reordering of signs and
symbols. This type of customization is supported by applications that allow users
to change the order of options in menus and toolbars, for instance.
Some (lexical) changes in the vocabulary may require corresponding (syntactic)
changes in the grammar, even if they do not introduce new meanings. For example,
more elaborate paraphrases saying that move le_F to directory_D is the same
as le_F moved to directory_D require that the lexical and the syntactic base be
expanded, although no new meanings are introduced. These are type III changes,
whose effect is to rephrase or paraphrase symbols and signs. An example of this
kind of customization is macrorecording (not macro programming). Strict macro
recording joins various lower-level sentences of the interactive language into a
sequential structure that is associated with a name. No new meanings are added
to the interpretive component of the application. Because users perceive this differ-
ently (i.e., they see it as a new meaningful element that is being added to the signi-
cation system that can be used to express and interpret communicative acts),
problems may follow. One of them is the encoding of constants and variables in
macro recording. Since recorded macros are no more than a rephrasing or para-
phrasing for existing interactive steps, important generalizations are not made
(generalizations that users often expect). So, many macro recording mechanisms
will encode the specic contextual values instantiated in the process being recorded
as part of the macro meaning. As a result, if a user is editing document1.txt and
decides to record the following steps:
Open File menu
Select Save as . . .
When dialog box shows up, select le type HTML
Press Save
168 Chapter 5
she may be surprised to see that when she runs the newly created macro to save doc-
ument2.txt in HTML format, the application will try to save the document as
document1.html, possibly in the same le directory in which document1.html
was saved. This is an interesting evidence of how macro recording is meant by users
and how it is handled by systems. By framing macro recording in terms of the types
presented in gure 5.2, designers can see more clearly the trade-offs involved in using
this type of technique.
Type IV changes, unlike the previous ones, explicitly introduce new meanings in
the original language specication. However, the syntax remains intact. This is the
case when new lexical items in the vocabulary are tokens of existing syntactic types,
systematically mapped onto semantic categories and interpretive functions. In a way,
this basic type of extension is present in all interactive applications that allow users
to create objects (e.g., documents, images, projects, and the like). Every time a user
saves a new document with any name he fancies giving it, the applications inter-
active code gains a new lexical item. Many applications explicitly incorporate such
recently created items in the File menu to accelerate access, a powerful evidence of
semantic expansion in the interactive code (even if transitory). Formally, type IV
changes can only expand lexical semantics (i.e., directly related to the meaning of
words, and independent of novel syntactic structures). Thus, in HCI this sort of
change is typically associated to the creation of application objects and falls within
the range of operations that the original application already does. Consequently, it
does not represent a particularly challenging possibility to explore in isolation. The
ability to introduce new lexical items expanding the syntactic and the semantic base
concomitantly is a much more useful resource.
Type V changes represent a complex linguistic manipulation of formal codes. By
using the same vocabulary, one can introduce meaning extensions signaled by dif-
ferent syntactic combinations and/or structures. An example of meaning expansions
achieved with new syntactic combinations would be to allow reordering to mean
different things. So if a particular interface can interpret the sequence Select_object,
select_tool, apply_tool_to_object, one can introduce a distinction by saying that
the combination
Select_object, select_tool, apply_tool_to_object
means that the tool should not be kept selected for the next step of interaction, but
the object should. The combination
The Semiotic Engineering of Customizable and Extensible Applications 169
Select_tool, select_object, apply_tool_to_object
would have the effect of preserving the tool selection and releasing the object selec-
tion for the next step. Notice that the actual dimension of meaning being manipulated
in this case is the pragmatic one, and not really the semantic one. It is the contextual
effect of using one phrasing or the other that is being changed, not the effect of the
tool on the object. But because these two things are typically handled together in com-
puter language specications, the semantic component is the one affected.
Syntactic manipulations in type V changes do not have to be conned to reorder-
ing. The creation of new structures can also be associated with the introduction of
new meanings. Keeping the context of the example we just gave for syntactic
reordering, let us suppose that now one wants to introduce the possibility of apply-
ing tools to multiple objectsa possibility that does not exist in the original design.
The application should handle structures like
Select_tool, select_list_of_objects,
apply_tool_to_list_of_objects.
This can be achieved by the introduction of list structures in the grammar, without
necessarily expanding the lexicon.
However, in most practical situations, type V changes are accompanied by some
kind of lexical change. If they arent, some strange inefciencies may arise. Let us
examine the expansion that allows users to select lists of objects. Without any kind
of lexical expansion in the vocabulary of the signication system, this new struc-
turing requires that every object be individually named in the list. Therefore, if there
are ve objects that the user wants to operate upon, he will have to say, for example:
Select_tool, {select_obj1, select_obj2, select_obj3,
select_obj4,
select_obj5}, apply_tool_to_{obj1, obj2, obj3, obj4, obj5}.
In the context of direct manipulation, for example, some expressive facilities
naturally accompany this possibility (e.g., using a bounding box to select multiple
objects in the same region). In the context of linguistic manipulations, other facili-
ties are used (e.g., using the combination Ctrl + A to select all objects, or com-
bining Shift + arrow keys to express continuous multiple selections). The bound-
ing box and the key chords are new lexical items that facilitate the codication of
new structures.
170 Chapter 5
When reordering introduces semantic distinctions, another sort of problem may
be found: higher cognitive loads for retaining semantic distinctions that are not lex-
ically marked. In the case of distinctive orderings causing the tool selection to persist
across different interactive steps or not, although this design choice is easily encoun-
tered in some applications, users are often confused by which combination does
what. To facilitate interpretation, accompanying vocabulary expansions such as a
new sign or symbol to express each of the two modes of operation (e.g., cursor
shapes assuming a form that refers to the selected tool when tool selections persist
over various object selections) are good alternatives. So for practical reasons, type
V changes usually give way to type VII changes in EUP or EUD circumstances.
Type VI changes introduce new semantics without affecting the grammar or the
vocabulary. This represents a radical meaning modication of existing signs and
symbols. If the modication is made directly in the semantic encoding of lexical
entries and syntactic structures, it changes the original meanings of design and may
thus potentially affect the applications identity. If it creates alternative interpreta-
tions, leaving the original ones intact, it introduces ambiguities in the interpretation
of signs and symbols. So from a computer perspective, type VI changes are a poten-
tial source of problems for extending encoded signication systems in extensible
applications, and should in principle be used very carefully, or not be used at all.
The situation is different when we take a users perspective, as will be seen shortly.
Finally, changes of type VII affect all dimensions of symbol processing. They
require considerable linguistic intuition and encoding abilities not easily encoun-
tered in the population of typical users. Such intuition and abilities are not those
applicable to the users own language, but to articial and formal languages, an
even more restrictive requirement. The kinds of techniques that enable such manipu-
lations on the originally encoded signication system are scripting and macro
programming. However, the difculty with these lies in controlling the preservation
of meanings associated with the impermeable signs that constitute the applications
identity. Suppose that the application we used to exemplify macro recording in the
type III changes mentioned previously behaved differently. Suppose that after
recording the Save as HTML macro, the user had the possibility of editing it, at
leisure, to the extent that she would be able to modify the dialog that showed up
in step (iii) and introduce in it the possibility of uploading the specied le to any
FTP site on the Internet.
This seems like a plausible idea that a skilled programmer, as a user, might perhaps
wish and be able to implement. The problem with this is that if the change is intro-
duced only in the context of the Save as HTML function, the lexical entry Save
The Semiotic Engineering of Customizable and Extensible Applications 171
suddenly becomes a confusing interface sign. The original Save cannot upload les,
but the extended one, appearing only in combination with HTML les, can. A
semiotic inspection will show that this represents a serious discontinuity in the sig-
nication system especially engineered for the application, in particular because
the meanings associated with Upload to FTP sites suddenly enter the users semi-
osic space. And there is no particular reason why these meanings would be unavail-
able for combination with other signs such as TXT les or PDF les, for instance
(as the sign Save is). So, in their generalized form, type VII changes present a serious
threat to the semiotic consistency of meaning extensions, which eventually may
break the systematic nature of associations between form and meaning in the appli-
cations interface. One way to control semiotic consistency is to constrain the lexical
and syntactic types that can be introduced by type VII changes. This is what is
done, for example, in the case of agent-assisted extensions (e.g., AgentSheets
[Reppening and Sumner 1995]) or programming-by-demonstration techniques (e.g.,
KidSim [Cypher and Caneld-Smith 1995]).
After having inspected the seven types of changes that may be explored to cus-
tomize and extend applications, we conclude the following from a computers
perspective:

There are two different classes of modications that users can introduce in applica-
tions: meaning-preserving and meaning-changing modications. We propose that the
former correspond to customizing applications and the latter to extending them.

There is a type of extension (type IV) that is part of almost every computer appli-
cation today. It allows users to introduce new tokens of prescribed grammatical
types whose interpretation can be automatically derived from token-type
systematic correspondences. So it is not particularly useful to include this type in a
taxonomy of changes that can be made in the original signication system of
customizable and extensible applications.

There is a type of extension (type V) that may result in impractical expression

requirements in HCI. In the absence of new lexical items that can express aspects
of syntactic and semantic expansions to the original signication system, users may
have to cope with inefcient structured expressions that use the same vocabulary
or with particularly heavy cognitive loads to retain mappings of distinct semantic
features onto syntactic structures that are not lexically marked.

There is a type of extension (type VI) that, if made, may either destroy the appli-
cations identity (which we propose should never be the case in EUP or EUD) or
172 Chapter 5
introduce undesirable lexical and syntactic ambiguities in the original signication
system. So, it is also not appropriate to include this type in our proposed
taxonomy.

Finally, there is a type of extension (type VII) that, because of its complexity,
requires further distinctions. Linguistic changes that affect the whole spectrum of
components in the encoding of signication systems may be conned to the manipu-
lation of only a subset of lexical, syntactic, and semantic types of items and rules
(VIIa), or they may extend to all types (VIIb). The usefulness of linguistic expan-
sions of type VIIa is directly proportional to the gains in expressive power enabled
by the strict manipulation of a selected subset of the lexicon, grammar and inter-
pretive rules. That of type VIIb expansions is directly proportional to the costs and
benets of checking that the desired modications do not destroy the applications
identity or introduce semiotic discontinuities into the expanded signication system
(a very costly procedure).
In gure 5.3 we summarize our conclusions. The shaded rows correspond to the
classes of changes that are either potentially problematic or semiotically impover-
ished. The implied suggestion is that the range of productive signication systems
manipulations enabled by user-dened customizations and extensions includes only
meaning-preserving lexical expansion or modication, meaning-preserving syntac-
tic modication, meaning-preserving structural expansions, type-controlled linguis-
tic expansions, and type-free linguistic expansions.
A frequently used adaptation that can generally be done at installation time
deserves further attention at this point. We will call it pruning, since it consists
The Semiotic Engineering of Customizable and Extensible Applications 173
Figure 5.3
Semiotic aspects in changing computer language dimensions.
of selecting items that are to be subtracted (or not added, depending on our
standpoint) from the complete application. For example, many text editors offer
users the possibility of choosing whether they want to install spelling and grammar
checkers. If users say yes, the signication system will handle all communication
necessary to use the checkers as texts are edited. Otherwise, the signs supporting
such communication will not (all) be part of the signication system, and probably
the semantic base will not include the meanings associated with spelling and
grammar checking. Because this kind of modication affects the semantic base, it
should be classied as an extension according to the denition we propose.
However, there are two problems with this classication. First, it is disturbing to
have an extension whose result is actually a retraction of the semantic base.
Second, and more important, because signs can be kept or taken out at installation
time, usually, it is disturbing to decide whether they are part of the applications
identity or not. In other words, are these retractable signs impermeable? If I say
they are, by my own denition, taking them out of the semantic base affects the
applications identity and amounts to letting users destroy the designers vision. Nev-
ertheless, it is the designer who is letting users retract the semantic base in this way.
Therefore, the applications identity must somehow be unaffected by such changes.
I thus suggest that impermeable signs come in two subclasses: essential and acci-
dental. Both are signs that cannot be modied by users during EUD activities.
However, accidental impermeable signs can be pruned (taken out) from the origi-
nal signication system, whereas essential impermeable signs cannot. And it is the
class of essential impermeable signs that constitute the applications identity.
Paying attention to pruning is important because the result of this type of
modication should not introduce semiotic discontinuities in the signication
system. So, for example, if users decide to keep the spell-checker but not to install
the grammar checker of a particular text editor, they should not nd themselves
involved in communication about noun phrases or capitalization (signs that belong
to the domain of grammar). One of the obvious design implications of this is that
interface dialogs must be carefully segmented. Signs corresponding to meanings that
have been subtracted from the semantic base at installation time should never arise
in communication unless the purpose of communication is to add them back to the
signication system.
The possibility of adding such signs back to the application suggests that the
semantic base is in fact not affected by pruning. That is, users can plug such signs
in and out as they wish, but the signs themselves are always part of the original sig-
174 Chapter 5
nication system. Consequently, pruning can be viewed as a meaning-preserving
operation in the sense that users are not denitively affecting the signication
system, but only pruning the set of signs that they wish to talk about with the system.
As a result, pruning comes closer to customization than extension, although it may
deserve more attention from designers than do other kinds of customizations.
5.2 Semiotic Aspects in Changing Signication Systems: The Human Perspective
From a users perspective, the relevant dimensions of codes are directly related
to the kinds of communication they enable. Intent is the dominant dimension in a
user-centered view of HCI. Semiotic engineering deals with intent in terms of
illocutionary and perlocutionary acts (see chapters 2 and 4). Illocutionary acts
(illocutions) characterize the speakers intent with respect to what he wishes to
achieve by expressing himself in one way or another (i.e., what he wants to [make]
happen). Perlocutionary acts (perlocutions) characterize his intent with respect to
the state of the world as a result of his expression (i.e., what actually happens in
situated discourse). Expression is the second outstanding dimension in a users per-
spective. It refers to the signifying code forms that can be used to achieve commu-
nicative goals (the users intent). Finally, content appears as an important dimension
in natural language communication. Expressions with different content are often
used to achieve the same intent. Such is the case of direct and indirect speech act
pairs like Would you mind closing the windows, please? and Its really cold with
the windows open, dont you think? Likewise, expressions with the same content
can achieve very different effects. Such is the case of ironic colloquial expressions
like It looks like youve been having a hard time, expressed in a context where
you refers to somebody who is being asked a series of difcult questions during
an examination, or alternatively to somebody who is sunbathing on a tropical beach.
In gure 5.4 we replicate the combinatorial possibilities that we used to analyze
the specic semiotic dimensions from a computer systems perspective. This time,
instead of having lexical, syntactic, and semantic dimensions, we have expression,
content, and intent.
There are again two classes of modications that a user may introduce in the
applications signication system: those that conserve the same scope of intent
accounted for by the original system and those that introduce new intentional ele-
ments. This subdivision should not, however, tempt us to replicate the customiza-
tion versus extension distinction proposed in the computer perspective. From a
The Semiotic Engineering of Customizable and Extensible Applications 175
human perspective, expression encompasses lexical and syntactic aspects.
Moreover, the semantic dimension in the computer perspective encompasses
content and intent dimensions in a human perspective. In gure 5.4, the only trace
of customization as previously dened is type A modications, which are but one
of seven potential modications that users might wish or need to make in the
encoded signication systems in order to suit their communicative purposes.
Intent-independent reencoding is an activity that adjusts the articial code without
explicitly aiming at introducing new intentional elements. Nevertheless, in the
process of unlimited semiosis, as soon as there are new expressions and content
there is automatically a chance to use them in order to achieve new intent. Let us
suppose that a user records the following macro:
Find Heading 2 format and replace it with Heading 3
format
Find Heading 1 format and replace it with Heading 2
format
Go to beginning of document
Type Title
Select previous word
Format selection as Heading 1
What this macro does is to add a new level on top of a two-tiered text structure.
From the computer systems perspective, the macro does not introduce novel seman-
tic elements. And, strictly speaking, from the users perspective, his perlocutionary
act can be achieved through the six steps he recorded as a macro named Push_head-
176 Chapter 5
Figure 5.4
Possibilities for changing communicative dimensions enabled by signication codes.
ings_down, for instance. So, we might argue that this constitutes an intent-
independent reencoding of interface signs (one that introduces a new expressive
alternative for preexisting content and intent elements). However, on the one hand,
if the user introduces a new expression, presumably this operation satises some
new dimension of the users intent. In this case, we can say that the users illocu-
tionary act is that the system will do the six sequential steps one after the other,
something that could only have been achieved through six individual illocutions
before the macro was created. On the other hand, the user might wish to apply
Push_headings_down two (or more) times in a row, in which case some nasty
effects would begin to appear (see gure 5.5). All lower-level headings would even-
tually be attened as Heading 3. So the introduction of new expressions in a signi-
cation system automatically magnies the scope of communicative goals that can
be achieved through the system, both from a strictly illocutionary perspective and
from a broader perlocutionary perspective. This is what can be seen in the illustra-
tive dialog at the beginning of this chapter, where Speaker 2 picked up Speaker 1s
metaphoric line and built new expressions and meanings on top of it. Thus, type
A, B, and C modications in an encoded signication system are somewhat
ctitious in HCI. Or, rather, viewing them in isolation from the users ongoing
semiosis may misinform some HCI design decisions relative to customizable and
extensible applications.
The preceding example helps us illustrate the kinds of type D expansions that users
might want to introduce in the encoded signication system of an extensible applica-
The Semiotic Engineering of Customizable and Extensible Applications 177
Figure 5.5
Effects of a recursive application of the Push_headings_down macro.
tion. The users intent, of course, is not to lump lower levels of the text structure all
together, but to extend the push-down operation to as many levels as there are in
the text. Thus, for instance, headings at level 3 would be pushed down to level 4,
headings at level 4 would be pushed down to level 5, and so on. The problem is that
macro recording per se is not the appropriate mechanism to achieve the desired degree
of generalization for the expression Push_headings_down. A quite elaborate pro-
gramming process would be required involving rst identifying of the lowest-level
headings, then pushing these down, and nally repeating the process until the topmost-
level headings were reached. The usual extension mechanism for this is scripting or
macro programming, which necessarily involves lexical, syntactic, and semantic
manipulations of the encoded signication system. Therefore, type D and type G
manipulations (where users are deliberately introducing new expressions, new content,
and new intent) involve the same degree of semiotic complexity from a computer per-
spective. Although users may not have such a ne degree of semiotic awareness to
explicitly classify what they want to do as type D or type G modications, they know
that recording macros is much easier than editing and programming them, and they
know that introducing the Push_headings_down expression in the encoded system
is different from introducing something totally new like Upload_le_to_my_website
(assuming that this function is not supported by the application). Nevertheless, from
the computer systems perspective, the difference may be no bigger than that between
a type-controlled and a type-free extension.
The remaining two types of modications in gure 5.4 are type E and type F. The
invariant in these two cases is that the expression used in communication is the
same, but the user means something else. Given the constraint we raised relative to
preserving the applications (encoded) identity, when speaking about the semiotic
dimensions in symbol processing, type E and type F modications should not allow
users to destroy the applications encoded identity. However, the user may well
repurpose some of the applications encoded signs. And this is the gist of type F
modications. An example of such repurposing would be if the user said Abort
application, but meant I am nished. Cast in typical interface signs, this is the
difference between nishing the dialog in gure 5.6 with the appropriate DOS
command, EXIT, for quitting the session, and doing it by typing Alt+F4 or click-
ing on the little x sign on the top right-hand corner of the window. Either one of
the three alternatives (each one with a specic semantic interpretation encoded in
the application) will result in nishing the conversation with DOS. However,
Alt+F4 is not the same as EXIT, and if meant to be the same it is in fact a case
178 Chapter 5
of repurposing Alt+F4 to mean what exit means, a strategy that, as we know,
will not always work well. An interesting aspect of type F modications is that they
are not modications at all from a computer systems perspective on customization
and extension. In this particular case, from the HCI designers perspective, they may
even constitute a mistake that will go unnoticed and be harmless in some cases,
but not in others. In a communicability evaluation of the application, moreover,
instances of such repurposing strategies are likely to be tagged I can do otherwise.
Thanks, but no, thanks would only signal extensions of this sort if the sign
used by the user to mean what he wanted was never deliberately encoded by the
designer as a sign to be used in association with the particular intent that the user
is trying to achieve. If the designer intended the sign to be used in association with
the users intent, but in other conversational circumstances, this is not a case of
repurposing the encoded meaning, but rather of disregarding the encoded rhetoric
and creating ones own idiolect (an individuals language use pattern) to interact
with the system.
Type E modications are the last to receive our attention in this discussion. They
innovate communication by allowing users to say something and mean something
The Semiotic Engineering of Customizable and Extensible Applications 179
Figure 5.6
A typical DOS window and command. Screen shot reprinted by permission from Microsoft
Corporation.
else (i.e., something other than the expression usually means). In the context of
extensible applications, the idea is that users are exploring expressive types enabled
by the existing forms associated to interface signs, but introducing innovation in
terms of content and intent. This strategy is what we naturally do when using
gurative speech, for instance. In natural language, if we say That poem is a pearl
we of course dont mean to say that the poem can be mounted on a ring and used
as an adornment. But we somehow mean That poem is a jewel, and that it has
a very special value. We are using a metaphor to express our opinion about the
poem, and at the same time opening an avenue of signication possibilities along
our metaphoric line. As has been shown in previous work (Barbosa and de Souza
2000, 2001), interpreting metaphors can be a useful mechanism for extending appli-
cations. Here is an example taken from Pine, a system that we used in chapter 4 to
illustrate contexts for using the Where am I? tag in communicability evaluation.
In Pine, the sign $ is sometimes used to sort the index of messages in a folder.
For instance, $ F has the effect of sorting the index of messages according to the
From eld; $ A sorts the index by Arrival Date; $ S sorts the index by
Subject; and so on. Now, when the user is working with the address book, there
are also elds in every address entry, namely, nickname, full name, Fcc, comment,
and addresses. But, if the user chooses the sign $ to compose an illocution to achieve
a sorting of address book entries, Pine tells him that command $ is not dened for
this particular context. So, if the user doesnt know it, he will have only made a
mistake. However, if the user knows that Pine originally cannot handle the sorting
of address book entries, but wants to introduce this possibility in communication,
$ Nickname becomes an intentionally innovative use of Pines command lan-
guage. The users expression is metaphorical in that it does not make literal sense
according to the interface code, but makes sense in an expanded scope of interpre-
tation where analogies and resemblances are the key to sense making.
Metonymical expressions might also achieve interesting effects. Suppose that a
novice Unix user has a directory named doc, in which there are les with exten-
sions txt and ps. Suppose, also, that this user for some reason wants to rename
the txt les in the doc directory as dat les. In version 4.1 of the Unix
command language, if from the hierarchical directory node immediately above doc
the user says mv doc/*.txt *.dat (meaning rename all txt les in directory doc
as dat les), the system will issue an error message (mv: missing le argument).
However, related commands where elements of the users expression appear are exe-
cuted without problem (e.g., mv doc/le1.txt doc/le1.dat, mv doc/*.txt ., and
180 Chapter 5
mv doc/le1.txt le1.dat). There are at least two important things that escape this
novice users understanding of how this operating system and its command language
work. First, that the mv command actually manipulates a les address in memory,
and not its name (although the subtle difference between these concepts is often dis-
regarded by nonnovice users). So, multiple les can only be manipulated through gen-
eralizing expressions such as *.txt if operations on each individual les address
have certain distributive properties (which mv happens not to have). Second, even
if mv *.txt *.dat could be interpreted as a DOS rename *.txt *.dat (i.e., if the
mv command could change the names of all the *.txt les to *.dat les, as it seems
to be able to do with individual les), the users expression mv doc/*.txt *.dat
would not necessarily mean (as intended) to rename txt les as dat les within
the doc directory. It could also mean (and most probably would, in a Unix context)
to move txt les from the doc directory to the current directory and rename
themas dat les. The purpose of this example is to show that the users meaning is
expressed through a classical metonymic expression*.dat is intended to mean
doc/*.dat and that metonymic expressions may, just like metaphoric ones, be
intentionally used to extend the range of illocutions (and perlocutions) that can
be communicated over a particular signication system. So, in conclusion, type F
modications in a users semiotic perspective allow us to explore the possibilities of
extending applications through gurative speech.
The result of our semiotic analysis from a users perspective is presented in gure
5.7, where shaded rows correspond again to cases that we are not considering in
our classication. Strictly speaking, all user manipulations of the signication code
are made to meet some kind of communicative goal (even if only as illocutionary
The Semiotic Engineering of Customizable and Extensible Applications 181
Figure 5.7
Interesting semiotic dimensions for customization and extensions from a human perspective.
acts whose effect does not require any expansion in the systems semantic base). As
a consequence, what we rst called intent-independent reencodings does not con-
stitute a useful class of possibilities in this perspective. The borderline between
customization and extension, from a users perspective, is also an articiality, which
reinforces some perspectives where both phenomena are treated in conjunction with
each other (e.g., the end-user development perspective [Lieberman, Patern, and
Wulf, forthcoming]). However, from the users point of view, there seems to be a
scale of increasing semiotic complexity in the way that an existing signication
system can be manipulated to introduce innovative communications. It ranges from
type F modications (repurposing of encoded expressions and contents), to type E
modications (gurative speech introducing new content and intent), to type D
modications (where users have to create new signs even if they will be reusing
the original signication systems semantic base to achieve their intent), and nally
to type G modications (a full-edged semiotic encoding, where the user has to
introduce new expressions, with their corresponding content and intent).
The message for HCI designers working on customizable and extensible applica-
tions carries a wealth of points for consideration. As suggested in gure 5.8, the
semiotic dimension and semiotic complexity correspondences between a computer
182 Chapter 5
Figure 5.8
Correspondence between human and computer semiotic dimensions and complexity.
(systems) and a human (users) perspective are somewhat disconcerting. What users
view as a single semiotic dimension bifurcates into two semiotic dimensions in the
computer systems perspective, and vice versa. Additionally, easy useful extensions
from a users point of view may be considerably difcult to achieve computation-
ally. Likewise, easy computational modications may require more elaborate semi-
otic operations from the user. Such mismatches in semiotic dimensions and
complexity may well explain why good EUD applications are so difcult to en-
counter, and why many of the encountered ones are so difcult to use (and
consequently not so good).
The benet of spelling out the semiotic dimensions and complexity involved in
customizable and extensible applications is to draw the designers attention to mean-
ingful design possibilities. The two levels of intermediary complexity from a users
point of view are the use of gurative speech and the rephrasing of expressions (to
achieve some illocutionary goals). Although they are inversely proportional to the
computational complexity of processing symbols in them, if the user had some
intuition of symbol-processing operations, these could be leveraged to achieve
interesting effects. In fact, that which we dened as customization activities from a
computer systems perspective suggests alternative ways to introduce, gently, symbol
processing. The very breakdowns that users so often encounter when trying to reuse
recorded macros in different contexts may give them a very good and strong intu-
ition of the difference between constants and variables in computation. Although
they will typically not be able to name these programming concepts as such, they
will certainly sense the difference between things that remain the same (although at
times they shouldnt) and things that take on different values all the time. The same
applies to breakdowns caused by underspecied contexts for recursion. They show
to the user that some systematic (and inferable) changes every time the same com-
putation is performed must be explicitly and thoroughly dened for a computer to
work properly. These ideas have been explored for some time in the eld of end-
user programming (DiGiano and Eisenberg 1995; Cypher and Caneld-Smith
1995). Programming by demonstration (PbyD) is one of the techniques used for this
purpose (Cypher 1993). But, as discussed in chapter 2, with respect to how differ-
ent categories of signs bring out different epistemic categories in the users mind,
PbyD may emphasize secondness over thirdness, and thus never lead users into
really understanding what programming is all about. The appropriate symbols,
which will bring out thirdness in programming, must not be terribly complex (Nardi
1993). Some applications have been explicitly developed to help users build such
The Semiotic Engineering of Customizable and Extensible Applications 183
computing intuitions. This is the case of StageCast
TM
Creator, for instance, which
uses a PbyD technique to help users specify very elaborate extensions to a series of
visually encoded signication systems. In gure 5.9 we see a screen of StageCast
Creator, showing how the new sign StepDown can be introduced in the original
signication system, touching on lexical, syntactic, and semantic dimensions. The
lexical dimension is dealt with by giving the new signicant element a name. The
syntactic dimension and its corresponding semantics are controlled by graphically
constrained representations of system states, before and after the desired operation.
This kind of technique requires elaborate inferential machinery that generates new
semantic elements from a combination of the lexical and syntactic manipulations
that the user species.
The other benet of spelling out the semiotic correspondences between a human
and a computer perspective is that designers may evaluate more clearly the costs and
benets of exploring more elaborate interpretive mechanisms (e.g., those that can
handle metaphoric and metonymic expressions). They can better understand the
users motivation or inclination to express himself in this or that way, or to expand
the encoded signication system to include new expressions, content, and intent.
184 Chapter 5
Figure 5.9
Screen shot from StageCast
TM
Creator 1.0. Screen shot reprinted by courtesy of StageCast
Inc.
5.3 Interaction as Problem Solving and Design: The Users Conversation with
Materials
Whereas traditional cognitive characterizations of HCI tend to cast interaction as
problem solving, a semiotic engineering characterization leads us to view HCI not
only as problem solving but also as design. Design is implicit (or explicit) in every
act of communication, once communicators start to make decisions about how to
use the available codes and modes of communication in order to compose a message
with which they expect achieve their particular goals. Problem solving in HCI (i.e.,
nding the answer to questions like How do I do this?) is preceded by problem
setting (i.e., nding the answer to questions like What is going on? and What
kinds of problems and opportunities do I have here?). The communicability eval-
uation method, presented in chapter 4, helps us see that problem setting is a crucial
stage in HCI. If users set the problem differently from what the original designer
expects, communication is more exposed to problems that can be characterized with
such communicability tags as I cant do it this way or I can do otherwise. Of
course, once the problem is set and framed, the problem-solving stage begins.
The similarity between Schns reection-in-action perspective on design (Schn
1983) and the semiosic processes involved in signication and communication sug-
gests that, like designers, users also have a kind of conversation with (semiotic)
materials. They can express an innite variety of intent (since every use situation is
different from every other with respect to the physical and the semiosic context of
its occurrence). If every intent were a different problem, then HCI would require
solving an innite variety of problems, with huge cognitive loads. But if every intent
is a different framing of a nite variety of problems, then a few interesting conse-
quences follow. First, human unlimited semiosis is accounted for by an innite
variety of interpretations that will determine what is going on and what kinds
of problems and opportunities the user has in every particular HCI situation.
Second, the limited interpretive capacity of computer programs is accounted for by
a limited set of semiotic innities, as we have put it in the beginning of this chapter.
Although interface codes allow users to say an innite set of things, there are other
innite sets of things that cannot be said (and/or interpreted). So, the users task is
to frame their intent as computable communication supported by the encoded sig-
nication system. Some framings will be successful, others will not. And the cate-
gories of unsuccessful framings will correspond to the categories of communicative
breakdowns presented in chapter 4. Third, if users are involved with problem-setting
The Semiotic Engineering of Customizable and Extensible Applications 185
(communication design) and problem-solving activities (decision making based
on rational or fortuitous criteria), supporting the users communicative design
activities brings HCI designers closer to designing customizable and extensible
applications.
As an example, an important piece of semiotic material that users can manipu-
late to design communication with systems is the set of interactive patterns encoded
by the original designer. Direct manipulation patterns such as dragging le icon f
onto application icon a is equivalent to applying the function a(f) become a
problem-solving method for every problem framed as function application. One of
the advantages of this perspective is to account for communicative situations where
function application is but one of the alternatives for framing the problem. The
other may be, for example, to specify a procedure (like open application a, create
an object o
a
, associate le f with o
a
, and terminate). A practical instance of
such different framings can be found in applications such as Winzip. Users can
achieve the effect of compressing a le in different ways. One is to drag the le icon
onto the Winzip icon, usually placed on the users desktop at installation time.
Another is to open Winzip, create a new archive, and add the le to the archive.
Both methods will result in a compressed version of the object le. Using the latter
is like saying Thanks, but no, thanks or I can do otherwise to the designers
expressed offer to facilitate the users interaction by allowing him or her to solve
the problem with fewer steps. When the communication of methods are as suc-
cessful as the communication of their relative advantage compared to each other
(one of the key contents in the designers deputys interactive discourse), the designer
is enhancing the communicative design environment that users will explore to frame
and solve interactive problems. And the kinds of content required to support the
users design help them understand the expressive possibilities and limitations of the
encoded signication system, which automatically advances some aspects of EUD
activities.
The kind of communicative design most frequently performed by users during
HCI is the design of interactive tokens. The encoded signication system enables
various types of expressions (see gure 5.1). When users design tokens of such
expressive types, they produce communication that falls into one of the three regions
inside the perimeter of the signication territory covered by the encoded system.
Some of the users tokens will trigger predicted and targeted perlocutions; some will
trigger predicted but not targeted ones; and yet others will lead to unpredicted and
nontargeted effects. So, for instance, successful interactions will usually be the result
186 Chapter 5
of the users designing the right tokens for targeted perlocutions. Likewise, interac-
tions resulting in specic error messages or warnings (e.g., Correct to: append/1?
in SWI Prolog), result from a faulty interactive token design (following with the
SWI Prolog example, listing(append/2).). And yet other faulty token designs will
trigger perlocutions that leave users rather clueless about what is going on (SWI
Prologs blinking cursor prompt following the sign | in response to the user enter-
ing the command listing(append/1), without the nal ., an example mentioned
in chapter 2). Are users expected to understand why the SWI Prolog interpreter can
guess that append/2 should be corrected to append/1 but not guess that
listing(append/1) should be corrected to listing(append/1).? Users who do
understand why the interpreter is prepared to handle one case but not the other
have ner perceptions about the various interactive types handled by SWI Prologs
encoded signication system.
Modications to the encoded signication system, from either a computer
systems or a users perspective, may involve the design of tokens and/or types. Types
are conceptually dened by generative rules. Therefore we may have lexical, syn-
tactic, and semantic types in computer codes. May we also have expression, content,
and intent types? Yes, indeed. They capture the original designers strategies for
encoding signs from the applications domain, from the proles of prospective users,
from typical activities and task models, and so on. In chapter 4 we identied six
basic models for which applications designers encode that which they expect to be
the users expression, content, and intent to be communicated through interaction.
The difference among lexical, syntactic, and semantic types is that expression,
content, and intent types are not usually available for computation as such, except
in intelligent user interfaces that can collaborate with users to help them achieve
their goals. In gure 5.10 we see how Microsoft PowerPoint 2002 captures a
type of interaction and associates it with a possible user intent. The user is clicking
on the crayons, but because they belong to another layer of graphics (the slide
master, in PowerPoint jargon) the application does not allow him to select the image
he sees. However, this pattern of interaction has been encoded by PowerPoint
designers as a signicant intent type encoded in the interface languagean attempt
to select an image from in the background. So an interface agent pops up and asks
the user if his intent is to manipulate images from the background layer.
The codication of intent types, as well as that of contents and expressions, is
used extensively by interface agents who are trying to help users communicate their
intent to the system. Making this codication more explicit would help users decide
The Semiotic Engineering of Customizable and Extensible Applications 187
if and how they want to extend or modify the range of communicative possibilities
allowed by the system. For instance, if the user is insistently clicking inside a polyg-
onal region from the top layer that corresponds to a polygonal region in the back-
ground layer where there is an object, the PowerPoint interface agent interprets that
the continuous mouse clicks mean that the user is trying to select an object from
the background layer. However, this intent type does not extend to a similar inter-
active pattern that might mean something similar, too. If the user insistently
clicks inside a region where there are no objects (in either the background or the
top layer), the assistant cant make any sense of it. Surprisingly, for many users, the
assistant does not suppose that the user is also trying to access the background layer
in this case, too (e.g., to change the ll color of the slide master). Why not? Why
doesnt it?
188 Chapter 5
Figure 5.10
Microsofts interface agent interprets an intent typeto select slide master graphics. Screen
shot reprinted by permission from Microsoft Corporation.
Because it doesnt, the users expression is an innovative form of communication
(with respect to the encoded signication system). Thus, here is a prime chance for
EUD. The user has to encode this kind of intent and do the appropriate mappings
from intent-content-expression to semantic-syntactic-lexical dimensions. This is
legitimately an act of design, and the more semiotic materials are explicitly avail-
able to the user, the easier his encoding task.
One of the great difculties for EUD in PowerPoint is that, although users can
record, edit, and create macros, EUD is not conceived (designed) as a semiotic
design activity. For example, if the user decides to build a macro step-by-step, cap-
turing the lexical and syntactic codication that corresponds to the expression he
wants to use (e.g., clicking insistently on nothing), he will have lots of trouble.
Recording insistent clicking yields no useful code in Visual Basic for Applications
(or VBA, Microsofts macro programming language) as can be seen in the follow-
ing transcription of the recorded insistent clicking macro:
01 Sub Insistent_Clicking()
02
03 Macro recorded 9/17/2003 by Clarisse Sieckenius de Souza
04
05
06 End Sub
In fact, the users strategy of using PowerPoint macro recording to gain access to
the systems encoding of the expression he wishes to compose with others in order
to communicate his new intent will fail altogether. Macro recording generates
object-oriented code in VBA. That is, expressive types are mingled with content and
intent types. For example, the code generated for clicking on the title text box in
gure 5.10 and dragging it down to the bottom of the slide is
01 ActiveWindow.Selection.SlideRange.Shapes(Rectangle1).
02 SelectActiveWindow.Selection.ShapeRange.IncrementTop 0.25
03 With ActiveWindow.Selection.ShapeRange
04 .IncrementLeft 114#
05 .IncrementTop 314.12
06 End With
The Semiotic Engineering of Customizable and Extensible Applications 189
Notice that we see no clicking and no dragging in the code, which is exactly
how the user expressed the action. But we see select and rectangle 1, and can
infer the semantic encoding of moving the text box as having to do with Incre-
mentLeft and IncrementTop values. Moreover, clicking events, like all mouse-
based interaction, must be associated with an object. Therefore, there cannot be
insistent clicking on nothing, which defeats the whole communicative strategy
conceived by the user.
This example reinforces the idea that EUP must not be such a big problem. The
problem is the programming language that users must know in order to do EUP
(Nardi 1993). The example illustrates one of the many kinds of mismatches between
a user-centered and a system-centered (or application-centered) perspective on EUD.
VBA is an incredibly powerful tool for programming extensions to a suite of
Microsoft applications, once the user adopts an application-centered perspective.
But it fails to deal with even the most basic semiotic intuitions that dominate a user-
centered perspective. It does not portray the applications signication systems in
terms of expression, content, and intent types that the user might wish to modify
and extend. Instead, it privileges the systems semantic dimension, even over the syn-
tactic and lexical ones, starting from an ontology of application objects, then moving
out toward certain types of behavior or states that encode syntactic and lexical
information related to what users perceive as expressive types.
5.4 Some Epistemic Tools for Users and Designers of EUP Applications
By viewing users as designers, semiotic engineering places a great value on epistemic
tools not only for HCI designers, but also for users. EUP applications are the best
example of how and why such tools may help designers achieve good metacom-
munication and users achieve more resourceful communication with the designers
deputy. As mentioned previously, every application has a unique signication
system, especially designed to convey the designers message. Even if the signica-
tion system resembles others, used in other applications, each is still unique because
its semantic base, where the applications identity resides, is unique. Therefore,
although Microsofts Ofce applications have similar and interrelated codes, for
example, the signication systems used in Word, PowerPoint, and Excel are not the
same. The same applies to Lotus SmartSuite. The signication system used in
Lotus Freelance Graphics resembles that of Lotus WordPro, Lotus 1-2-3,
and other applications in the suite, but they are not the same. We would never think
190 Chapter 5
that Freelance belongs to Ofce, or that Excel is part of SmartSuite, even if each
individual application has its unique signs.
Resemblance among signication systems is of course a very powerful resource
when users must learn a unique system for every unique application they wish to
use. Having general design patterns instantiated (with the necessary adaptations)
across classes of applications helps users infer and memorize what signicant expres-
sions mean or may mean in various contexts of use. But the rst step toward pat-
terned design is to decide (a) what to design and (b) why. In semiotic engineering,
the answers are, respectively, (a) to design a signication system for communicating
the designers vision effectively, because (b) productive use of the designed artifact
depends on the success of the designers communication. Thus the selected design
patterns must meet communicative needs encountered by designers at design time
and by users at interaction time.
The gist of the designers metacommunication message in customizable and ex-
tensible applications should let users understand or infer the scope and effects of
possible changes. The paraphrasing of the message, with specic reference to how
customizations and extensions must be expressed, is thus the following:
Here is my understanding of who you are, what Ive learned you want or need to do, in
which preferred ways, and why. This is the system that I have therefore designed for you,
and this is the way you can or should use it in order to fulll a range of purposes that fall
within this vision. But, I know you may want to modify my vision in order to do things (in
a way) that I havent thought of. I can handle the changes you may wish to do, provided
that you can say what you want in this particular code.
In chapter 4, we proposed that the designers deputys discourse is composed of
elements selected from six different models of the application: the domain model,
the user model, the application model, the task model, the interaction model, and,
nally, the interface model. The designers deputys discourse is then encoded in the
applications unique signication system, which means that the semantic base of the
system refers to the elements of these six models and to the relations among them.
So we can expect to see semantic types in the semantic base. For example, an appli-
cations functions constitute a type, just as an applications objects, objects attrib-
utes, and so on do. Such types help users internalize unique signication systems
faster and more effectively. Likewise, interaction types like multiple and single selec-
tion of objects, activation of functions and procedures, naming and discriminating
elements linguistically, all help users grasp the design vision and anticipate how the
signication system can and must be used in communication with the system (i.e.,
The Semiotic Engineering of Customizable and Extensible Applications 191
the designers deputy). Interface types (which are not the same as interactive types)
also help users communicate more efciently. Representing interactive types with
constant interface types like the same kind of menu entries, or spatial relations, or
font, reinforces the users learning and facilitate recall and abduction. For instance,
an interactive type like this, that, or forget, three-way mutually exclusive choices
where the rst two are complementary alternatives and the third is a deferral, can
be conveyed through different interface types. In gure 5.11 we see three of them.
On the left-hand side, types are instantiated to different sets of values. On the right-
hand side, types are depicted without values but with the signicant meaning fea-
tures that determine the order of choices.
Typed design, with respect to each and every model involved in building the
designers deputys discourse, will not only facilitate learning and productive inter-
action, but considerably help frame and solve customization and extension prob-
lems. One of the most difcult issues in EUP, from a semiotic engineering
perspective, is to motivate extensions that will not introduce semiotic discontinu-
ities into the original system. For example, if the applications designer has encoded
this, that, or forget interactive types as interface types like (c) in gure 5.11, allow-
ing the user to encode an instance of this, that, or forget with interface type (b)
will introduce a discontinuity in the original system. Although the discontinuity may
be harmless in most situations, it may gain importance if (b) has been used origi-
nally by the designer to convey mutually inclusive choices among as many alterna-
tives as there are. In this case, after the users extension, interface type (b) (a list of
items) will mean two different things. In some cases, it will mean choose as many
as you like from these, and in other cases it will mean choose the most likely or
the less likely alternative, or defer to make a choice.
192 Chapter 5
Figure 5.11
Alternative interface patterns for expressing the same interactive pattern.
Typed design is a by-product of model-based design, but software models are
widely believed to be too time-consuming and to offer few rewards in professional
use situations. So design types are usually those derived from professional practice
and organizational standards. But they are not necessarily the ones that will allow
designers to establish explicit semiotic connections between software model com-
ponents and lexical, syntactic, and semantic dimensions of the original signication
system. A natural resistance to spending lots of resources when benets are unclear
has been perpetuating HCI beliefs and solutions that may, and perhaps must, be
replaced. And there are already signs of a growing awareness that unless there are
radical changes in the way we think about computers and systems, users are not
likely to experience any signicant improvement when interacting with computers
(Dourish 2001). The semiotic engineering contribution to the ongoing debate is to
connect some points about model-based design that are usually considered in iso-
lation or in a narrower scope (and therefore weakened). Design models, enriched
by semiotic connections, can

improve metacommunication (through an enriched designers deputys discourse,

as seen in chapter 4); and

facilitate EUD by supporting signication system extensions through

communication using (certain kinds of) gurative speech;

communication expressing explicit type-controlled metalinguistic operations; and

communication with fellow users or developers about (expected or effective)

extensions.
5.4.1 Extending Software through Metaphors and Metonymies
Extending software through metaphors and metonymies is an EUP technique that
operates on explicit semantic models of an applications signication system
(Barbosa 1999). As designers encode elements of various underlying models, they
must explicitly encode metaphoric and metonymic relations that they see among the
elements of the signication system. For example, an examination of Pines signi-
cation system may lead designers to see some analogies between elds appearing in
a listing of inbox messages and elds appearing in a listing of the address book.
Saying that both types of elds are alike will enable metaphoric interpretations of
incorrect sort commands in Pine, such as $ nickname. The command may then
be interpreted to mean do with nickname in the address book context what you
would do with a eld in the inbox context (i.e., sort addresses by nickname).
The Semiotic Engineering of Customizable and Extensible Applications 193
This kind of interpretation depends on the implementation of abductive reasoning
algorithms that will compute upon metaphoric (and metonymic) relations explicitly
encoded by designers (Barbosa and de Souza 2000, 2001). Metonymic relations
typically establish connections between part and whole, container and contained,
product and producer, and so on.
When abductive reasoning leads to more than one plausible interpretation, com-
munication with the user helps disambiguate them. An explicit encoding that elds
in the inbox and elds in the address book are alike (metaphor) and that items in
a folder are contiguous to the folder itself (metonymy) can yield interesting results.
If the user is viewing the details of an address entry and says I (I is the command
to view the index of messages in the inbox), the interpreter may conclude one of
two things. That I means Do in the address item view context what you would
do in the inbox item view context (i.e., show the index of other entries in the
parent folder, the address book in this context), or that I means Do in the address
item view context what you would do in the whole address book view context
(i.e., show the index of messages in the inbox). The overall effect of processing
metaphors and metonymies is to enlarge the communicative possibilities over the
same signication system and to achieve more forgiving HCI (where the notion of
error is considerably relaxed). Moreover, some of the metaphoric and metonymic
encodings may be permanently added to the signication system, if the user so
wishes. For example, $ nickname may become what we call in communication a
dead metaphor. It becomes a regular grammatical sentence whose interpretation is
systematically associated with it every time it is used in communication. This is pre-
cisely how EUP may be achieved through interpreting certain types of gurative
speech. The additional components are the abductive reasoner and the metaphoric
and metonymic relations in the semantic base.
5.4.2 Extending Software through Explicit Type-Controlled Metalinguistic
Operations
Extending software through explicit metalinguistic operations is usually achieved
by macro programming or scripting. The border between extensions and full-edged
programming is not clear, since users can produce programs and scripts that intro-
duce arbitrary modications in the original applications signication code. In our
rst approach to controlling the types of modications that users should be allowed
to make, we proposed two principles applicable to the original systems signica-
tion code and to the metalanguage (the scripting or programming language) through
194 Chapter 5
which extensions are specied: the interpretive abstraction principle and the semi-
otic continuum principle (de Souza, Barbosa, and da Silva 2001). The principles
were proposed for evaluating EUP. The interpretive abstraction principle aims at
assessing how well the signication system abstracts and encapsulates the applica-
tions semantic base (original or extended), whereas the semiotic continuum princi-
ple assesses the obstacles in communicating extensions through an extensible
signication system. In other words, the former takes an interactive perspective and
asks What do (new) interactive signs mean?, and the latter takes an extension
perspective and asks How can or should these extensions be signied for interac-
tion? The principles will help users nd the answers to these questions more easily
than they do in most commercially available extensible applications.
Given a signication system S, whose semantic dimension is specied in one or
more languages L, S signs (simple or compound) are interpretive abstractions of L
signs (simple or compound) if users can make sense of all S signs by resorting only
to

other signs encountered while interacting with the application where S is encoded
(this includes the designers deputys discourse about the application); and

their own cultural background (including knowledge of the applications domain,

computer literacy, and common sense).
The interpretive abstraction principle is expressing general HCI knowledgethat
interface signs should not require from users other knowledge than that introduced
by the application itself and the knowledge the user already has about his world.
However, it explicitly says that the ingredients in the users semiosis are the signi-
cation systems signs (in particular, the signs of the designers metacommunication,
unfolded by the designers deputys communication about the application) and the
cultural signs with which the user is familiar. In other words, both types of signs
will merge to contribute to the users sense making while communicating with the
system. This principle also states that if signs from the semantic base languages
(signs of L) inadvertently surface in communication (e.g., in error messages like
index out of bound), the signication system is breaching the principle. As a
result, the designers attention is turned to the importance of framing the whole
system as the object of design, and not only parts of ita practice encouraged by
some compositional design techniques that place a high value on reuse.
The semiotic continuum principle states that any two languages, L1 and L2, are
semiotically continuous if
The Semiotic Engineering of Customizable and Extensible Applications 195

L1 is an interpretive abstraction of L2;

L2 is an interpretive abstraction of one or more languages;

L2 has a specic syntactic structure equivalent to texta compound sign with

illocutionary and perlocutionary dimensions whose values are established accord-
ing to users goals and systems functions, respectively; and

any user who knows L1 and L2 can always translate L2 texts into actual or poten-
tial signs in L1.
An actual sign in any language (or signication system) is an encoded sign in the
language. A potential sign is a nonencoded sign of the same (lexical, syntactic, or
semantic) type as other existing types in the language (or signication system). For
example, a potential interface sign is one that is designed according to existing
design types. Thus, wod is a potential sign in English, but wrsit is not. Like-
wise, Save as DAT . . . is a potential sign in most commercial text editors le
menus, whereas Save as 50% is not. So what the semiotic continuum principle
states is that the underlying model of signication systems (and types therein) should
be known to users (intuitively or as a result of learning, with or without the
help of assistants) if users are expected to manipulate one system to generate
symbols in the other. Moreover, the semiotic continuum principle accounts for some
of the innovative uses of signs in communication, according to Ecos TSP (Eco
1976). It is a clear case of users communicating signs that are not present in the sig-
nication system itself, but considering the existence and the logic of the signica-
tion system.
Because it deals with the pragmatic notion of text, the semiotic continuum prin-
ciple can also prevent some common mistakes in EUP. For instance, if the scripting
language and the interface language are semiotically continuous, scripts that do
nothing (i.e., that dont achieve any kind of perlocution) or that have no corre-
sponding interface sign to trigger their execution (i.e., they cannot be accessed by
the user during interaction) may be syntactically diagnosed as faulty (because the
text structure is absent from the script). Likewise, existing semantic types (to
account for the users content and intent), as well as syntactic and lexical types (to
account for the users expression) in the signication system, may be made avail-
able to users through various techniques. There may be templates in a database of
scripts, which users may query by types of content, intent, and/or expression. There
may also be scripting assistants to guide users engaged in building extensions. Even
gurative speech interpreters may be used to infer the users text from a literally
196 Chapter 5
incomplete or incorrect text in the script. For all of them, there are two important
components in the applications architecture, from which semiotic continuity can be
evaluated: a design rationale knowledge base and an intelligent extension interpreter
(that can check extensions against the knowledge base, answer questions about the
applications design rationale, and include knowledge about users extensions in the
base) (da Silva 2001). The knowledge base must, of course, have explicit represen-
tations for all design types, as well as the semiotic connections between such types
and the range of actual or potential expressions, content, and intent that users can
communicate. This kind of component is very similar to the ones that are used
in programming by demonstration, for example. But in explicit metalinguistic
operations, the kinds of extensions achieved by users may be different from the
ones achieved by programming by demonstration (where the connection between
interface language and scripting language is much tighter than in linguistic
translation).
5.4.3 Supporting Users Communication about Extensions
Another advantage of model-based design where connections between the users and
the systems semiotic dimensions are explicitly marked is that users can begin to
communicate with each other and with developers about extensions (and cus-
tomizations) to the existing signication system. Andersen (2001) has used the term
compulsive talkers to characterize the fact that we continuously incorporate our
experiences in communicative discourse. As a consequence, one of the desirable
qualities of applications interfaces (and HCI as a whole) is that they must be ver-
balizable (Andersens term). In other words, users must be able to talk about them.
Semiotic engineering has a strict interpretation of this position and proposes that
the designers deputy must be able to talk about the application with users, in order
to convey the designers metacommunication message. That is, the signication
system itself must incorporate verbalizations of the users experience, which
will appear in the designers deputys discourse. The semantics of such verbalized
expressions is to be found, again, in the various design models generated by the
designer in the process of producing an artifact that expresses his complete design
vision.
One of the benets of this is that users can more easily communicate with each
other (user groups) and with developers and designers about their doubts and sug-
gestions for modications in the product. Thus, positive impacts on the software
development cycle are likely to be felt. Moreover, as users begin to talk with each
The Semiotic Engineering of Customizable and Extensible Applications 197
other about the application they share, they can begin to produce collaborative cus-
tomizations and extensions, a highly desirable possibility in the context of group-
ware applications that must be exible enough to support group dynamics in
ever-changing contexts (Dourish et al. 1996; Jrgensen 2001). Experience with spe-
cic signication systems encoding different models involved in the design of work-
ow system (Cunha 2001) has shown that users nd interesting ways to express
their vision of applications extensions. Manipulations of signs appearing in inter-
face layouts, networked representations of systems processes, and information
structure diagrams have all been used to communicate the users intent. Cunhas
point is that if all such representations are explicitly articulated in a metalanguage,
some types of extensions can be achieved by users themselves without intervention
of developers. This is an important step in toward adaptable multi-user applications
in general, where meaning negotiations are more complex than in single-user envi-
ronments, and may therefore lead to more complex design problems, as we will see
in the next chapter.
5.5 Epistemic Tools for the Design of Customizable and Extensible
Applications
This chapter introduced epistemic tools that can help designers build customizable
and extensible signication systems. It began by portraying software customization
and extension as a means to approximate human semiosis, and thus to enable a
more natural kind of communication between users and systems. This conceptual-
ization led to an analysis of semiotic dimensions involved in symbol-processing
activities performed by computer systems and in human communication performed
by users. Two important epistemic tools were then proposed. First, a taxonomy of
modications that can be made to an engineered signication system, both from a
systems and a users perspective, allowed us to make some important distinctions.
The taxonomy is not the same for systems and users, and technical distinctions
between concepts such as customization and extension, although practical from a
designers point of view, have been shown to have no parallel in the users experi-
ence. The taxonomy also suggested that different classes of modications corre-
spond to different degrees of semiotic complexity for the computer or the human
being. So the second epistemic tool in this chapter was a set of correspondences that
we can try to establish between the semiotic dimensions proposed for computers
and humans. The result of such attempt shows that there is no one-to-one corre-
198 Chapter 5
spondence between them. On the contrary, correspondences are disconcertingly
irregular or ambiguous, which helps explain why applications supporting EUD are
so difcult to design and/or use. Nevertheless, as difculties and complexity are
spelled out, some interesting possibilities arise.
This chapter has again emphasized the fundamental role of design models in
enhancing communication between users and computers, and in arguing for such
models it has taken a few steps forward in identifying some interesting consequences
of model-based design when semiotic connections between computer and human
dimensions are established. In particular, semiotically enriched model-based design
can support two types of EUPextensions through gurative speech and through
collaborative extensions. These possibilities show that model-based design may not
only appeal to developers (who hope to achieve faster and better software devel-
opment, resorting to automation, reuse, consistency, and other qualities), but also
open new interactive possibilities and enhance the users experience in interesting
qualitative ways.
Finally, this chapter presented two principles that can be used to analyze lan-
guages, or signication systems, when the meanings of one are dened or expressed
in the other. The interpretive abstraction and the semiotic continuum principles can
be used to explain (and in certain circumstances to anticipate) difculties encoun-
tered by users when building linguistic specications of extensions to an existing
signication system. The explanatory power of these principles can help designers
understand semiotic aspects of EUP and thus frame and solve problems in such a
way as to improve the users experience while trying to tailor applications to their
specic needs and wishes.
The taxonomies, relations, and principles presented in this chapter may be used
to analyze existing customization and extension techniques and to explore novel
ones. They help designers view EUD in various ways and thus organize the design
space according to this view. An example what novel space organizations can do,
as I suggested, is to view sign repurposing (or individual nonstandard use of inter-
face signs in communication) as a kind of customization, and not just as a mistake.
In connection with communicability evaluation, this kind of phenomenon is telling
designers that software artifacts are indeed semiotic material for users to design
communication as they can or wish. And it places a special value on encoding task
models, user models, domain models, and so on, not as a prescriptive instrument
for what users must do, for what they are, or for what they know, but very much
as a meaning-negotiation instrument for letting them know what designers were
The Semiotic Engineering of Customizable and Extensible Applications 199
thinking about all of this. Having this perspective on software design may even help
users become codesigners of software, because they can begin to talk to developers
about such meanings and their encoding. So the implications of semiotically
enriched design may be far-reaching, which is my main motivation in organizing
semiotic engineering as a theory that can be studied and critiqued before the knowl-
edge it generates can take the form of design methods and techniques or computer-
aided design tools.
200 Chapter 5
6
The Semiotic Engineering of Multi-User
Computer Applications
Advances in technology have popularized computer applications where users inter-
act not only with a system but also, and more important, with other users. The
purpose of interaction is widely variedfrom collaborative work to entertainment,
from socializing to professional or emotional supportand can be expected to
follow human imagination wherever technological possibilities may lead. Termi-
nology such as CSCW, computer-supported collaborative learning, CMC, online
communities, virtual communities, communities of practice, e-groups, multi-user
dungeons, and groupware illustrates some of the many different perspectives taken
by researchers and technologists to explore computers as a medium for human
activity.
We refer broadly to all kinds of software that aim at supporting or enabling
human interaction online as multi-user applications (MUApps). Whether interac-
tion is achieved through synchronous or asynchronous communication, with heavy-
weight (e.g., virtual reality) or lightweight (e.g., discussion forum) technology, in
stationary or mobile equipment, the semiotic engineering of MUApps benets from
epistemic tools presented in this chapter.
6.1 Three Conceptual Metaphors for Computer-Mediated Communication
The design of MUApps has been extensively inspired by three conceptual metaphors:
the communication center metaphor, the virtual environment metaphor, and the
telecommunications device metaphor (see gures 6.1a,b,c). Two decades ago, Halasz
and Moran (1982) warned designers about the harmful effects of teaching users the
analogy instead of the applications conceptual model. Analogy, the authors said,
should be used as literary metaphor (385)a means for expressing new content,
not a content in itself. However, as Lakoffs research (1987) in cognitive semantics
has shown, the boundaries between expression and meaning are not as clear as
Halasz and Moran would wish. The use of metaphors motivates certain semiosic
paths rather than others and thus affects cognition in important ways. The func-
tion of metaphors is not only to simply illustrate individual properties of objects
(Halasz and Moran 1982, 385). Rather, human conceptual categories have prop-
erties that are a result of imaginative processes (metaphor, metonymy, mental
imagery) that do not mirror nature (Lakoff 1987, 371) and things as they are. The
202 Chapter 6
Figure 6.1.a
The system as a communication center.
Figure 6.1.b
The system as a virtual environment for interaction.
Figure 6.1.c
The system as a telecommunications device.
users learning of conceptual models is always affected by mental imagery spring-
ing from various sources: individual experience in the world, culture, the history of
situated interaction with the application, the signs present in the applications inter-
face, and so on. Hence the interest of exploring expression and meaning involved
in these conceptual metaphors for computer-mediated communication.
The fact that these basic metaphors are present in most existing MUApps con-
tributes to the formation of a microculture that plays an important role in how users
categorize and understand concepts involved in computer technology to support
group activities. A single application may incorporate more than one metaphor, and
one of the important design decisions to be made in this case is how to articulate
metaphors with one another so that users are not disoriented in categorizing mean-
ingful elements of a conceptual model as it takes shape.
6.1.1 The Communication Center Metaphor
The communication center metaphor consists of an interpretive and expressive
schema in which the MUApp is cast as a central service provider attending to the
users requests for communication. Various services are presented to the users in
specic forms, which determine how requests are made. Figure 6.2 depicts a screen
The Semiotic Engineering of Multi-User Computer Applications 203
Figure 6.2
An instance of the communication center metaphor in MSN Messenger 5.0. Screen shot
reprinted by permission from Microsoft Corporation.
shot of MSN Messenger, where users requests are expressed in natural language
sentences starting with I want to . . . This feature underlines the communication
center metaphor, since the system acts as an attendant. The implicit conversational
style (where the attendant is supposedly asking What do you want to do?) carries
important anthropomorphic (and social) ingredients compared to the one adopted
by other MUApps, where services are expressed simply as chat, email, tele-
conferencing, le transfer, discussion forum, and so on.
The metaphor may be expressed more weakly, as a mere evocation of the
concept(s) it carries, or more strongly. Characterizing the system as an attendant
that users talk to in order to achieve various communicative goals may go as far as
incorporating human-like interface agents. Figure 6.3 shows a screen shot of
204 Chapter 6
Figure 6.3
Microsoft Assistant helps Outlook user change meeting attendants list. Screen shot
reprinted by permission from Microsoft Corporation.
Microsoft Outlook, where Merlin (the interface agent) is trying to help the user
change the list of attendants for a particular meeting. The interface metaphor leads
one to suppose that the agent can understand natural language and take intelligent
action. Inasmuch as agents like Merlin are in fact able to help the users directly
achieve their mutual communication goals, the communication center metaphor is
strongly enforced.
6.1.2 The Virtual Environment Metaphor
The virtual environment metaphor is an interpretive and expressive schema where
designers seek to promote immersive or semi-immersive experiences. The possibili-
ties for interaction and activity are enabled by the structure of the virtual space
where users are projected. The structure enables and suggests certain types of inter-
action, while inhibiting others. The designers goal is to promote (semi-)immersive
experiences that will help users learn certain conventions embedded in the applica-
tion and develop their own conventions for interacting with other users. The role
of the virtual environment metaphor is to stimulate users to imaginatively project
from certain well-structured aspects of bodily and interactional experience to
abstract conceptual structures (Lakoff 1988, 121), to cast their knowledge and
practice of face-to-face contacts with individuals and groups in the MUApp envi-
ronment. A good illustration of this type of design metaphor is Active Worlds

.
Figure 6.4 features the welcome page of this 3-D chat. In the center of the page is
the virtual environment where various users are chatting. Each user has his or her
own representation (i.e., each user is a sign). The user moves around the world,
meeting and talking with people (a projection of physical and social experience).
The frame on the right-hand side of the screen tells the newcomer how to move
around, thereby emphasizing the designers intentthat users will experience
Active Worlds as a virtual environment where they can project noncomputational
reality.
Anticipating the contextualized meaning and effect of direct manipulation signs
(well-known to the computer literate) in semi-immersive MUApps may be an impor-
tant initial barrier. For instance, what happens if a user clicks on the representa-
tions of others? What if he or she tries to drag and drop them around the virtual
space? The environment itself usually contains other manipulable objects. For
instance, in the context of a virtual 3-D bookstore, clicking on a book may mean
Let me see this book. Dropping the book in a shopping basket may mean This
is a book I intend to buy. If the book is taken to a checkout counter, this may
The Semiotic Engineering of Multi-User Computer Applications 205
mean I am buying this book. The difference between manipulating representa-
tions of objects and representations of people is, however, considerable.
People have full-edged cognitive and semiotic capacity. Their interpretation of
computer-mediated interaction with others may bring up feelings of joy and pleas-
ure, as well as discomfort and offence. So, the inherent challenge of designing
MUApps that follow the virtual environment metaphor closely is to provide users
with enriched signication and communication systems that will allow them to
project their knowledge of the world onto computer representations in order to tell
and do to others exactly what they mean (or as nearly so as possible).
206 Chapter 6
Figure 6.4
An instance of the virtual environment metaphor in Active Worlds. Screen shot reprinted
by courtesy of Activeworlds Inc.
As already mentioned, MUApps seldom match a single metaphor. Active Worlds
combines the virtual environment and the communication center metaphor,
for example. It provides an advanced mode interface, and users are warned that
this interface can be confusing to novices. One of the features in such advanced
mode is that there are side tabs on the left-hand side of the screen representing
telecommunication services (like contacts or telegrams). In general, applica-
tions that offer a wider variety of functions will tend to contain at least a
portion that is designed according to communication center metaphor, because it
explicitly interposes an intermediary (that can occasionally provide explanations,
prompt for more details, and/or treat exceptions) between user and services. This
intermediary corresponds exactly to the conversational representation of the
designers deputy. The virtual environment also speaks for the designer, by means
of a reied discourse that is considerably constrained compared to conversational
discourse.
6.1.3 The Telecommunications Device Metaphor
The telecommunications device metaphor consists of an interpretive and expressive
schema where the MUApp is cast as a machine or mechanism, in an emphatically
utilitarian perspective. Users can wield the MUApp as a physical tool in order to
communicate with others. A good example of this perspective is provided by Ring-
Central PhoneWorks fax (see gure 6.5).
Whereas in Messenger (gure 6.1) users request access to email or voice conver-
sation using a dialog style (I want to . . . Send Email or Start a Voice Conversa-
tion), in PhoneWorks users directly manipulate the controls on the device. One of
the major design challenges with the telecommunications device metaphor is to keep
the design simple, intuitive, and efcient. As more functionality is added to the
system, more information is needed, more parameters can or must be set, more addi-
tional devices and systems tend to be connected and integrated to a particular piece,
and so on. Consequently, maintaining the metaphor throughout the whole range of
interaction possibilities is very difcult. In gure 6.6 we see that PhoneWorks uses
an integrated address book, which is not a telecommunications device itself,
but a memory aid that is typically used with telephones and fax machines. The
design metaphor holding for the machine naturally extends to connected objects,
such as address books and video cameras, for example. But as the number of con-
nected objects increases, one approximates the virtual environment metaphor.
Nevertheless, there is a crucial distinction between the two metaphors: in
The Semiotic Engineering of Multi-User Computer Applications 207
telecommunications devices the self and others are represented only in linguistic
form (e.g., by their names).
The physical and conceptual identity of address books and other virtual objects
is substantially transformed and expanded in MUApps. Notice in gure 6.6 that
RingCentral advertises PhoneWorks integration with other applications that can
directly dial the numbers recorded in the address book. This possibility is not present
in physical address books that contain only references, but not causal links, to phone
208 Chapter 6
Figure 6.5
PhoneWorks photorealistic interface. Screen shot reprinted by courtesy of RingCentral Inc.
(http://www.ringcentral.com).
numbers. Even street addresses may be linked to geographic positioning systems and
online maps, which again transforms the nature of linguistic signs into something
of an interactive experience. All these possibilities arise from intertwining the
primary telecommunications device metaphor with other technological and cultural
signs, some of them expressed as devices themselves.
6.2 Designer-to-Users Communication
The metacommunication message sent from designers to users in MUApps is the
most elaborate case of all similar messages analyzed in this book. It contains the
basic single-user message repeatedly invoked in previous chapters, but it also
includes important additional elements. The gist of the metacommunication message
for MUApps can be paraphrased as follows:
The Semiotic Engineering of Multi-User Computer Applications 209
Figure 6.6
Virtual address book integrated to PhoneWorks extends the capabilities of its physical equ-
ivalent. Screen shot reprinted by courtesy of RingCentral Inc. (http://www.ringcentral.com).
Here is my understanding of who you are, what Ive learned you want or need to do, in
which preferred ways, and why. And this is the system that I have therefore designed for you,
and the way you can or should use it in order to fulll a range of purposes that fall within
this vision. You can communicate and interact with other users through the system. During
communication, the system will help you check:
Who is speaking? To whom?
What is the speaker saying? Using which code and medium? Are code and medium
appropriate for the situation? Are there alternatives?
Is (are) the listener(s) receiving the message? What if not?
How can the listener(s) respond to the speaker?
Is there recourse if the speaker realizes the listener(s) misunderstood the message? What is
it?
The metacommunication message is highly nuanced. The MUApp designer is
stating that users can nd answers to the ve groups of questions in the message.
But good answers require good design. For example, some MUApps try to allevi-
ate the communicative burdens of work group members by detecting certain com-
municative events and sending messages on behalf of users. In many situations these
messages may be useful and help sustain the efciency of collaboration. Let us
suppose that one member is chatting with another member who suddenly loses con-
nection. The system then automatically speaks for the member in trouble and
replies to the senders last message by saying I am having trouble with my
connection. Please, wait. The positive aspect of this message is that the user is
immediately aware that chatting with this colleague is a problem at that moment.
But, although the system intervention may apparently score high on usability and
sociability scales, it prevents the user from nding the correct answer to the very
rst question in the designer-to-users metacommunication message. It has a lower
score on communicability scales. Who is speaking? The system message is so mis-
leadingly phrased that the sender of the rst message may well go on and say
Maybe I can help you. Whats the problem? If his (presumed) interlocutor repeats
I am having trouble with my connection. Please, wait, he may well think the
person is being rude (certain angry intonations of the sentence make this clear). The
longer it takes to the sender to realize that the legitimate receiver is out of reach
and that the system is intercepting communication, the greater the chances are that
a breakdown will occur. When users eventually realize that the system always speaks
in this way for members who are temporarily disconnected, they will probably start
to wonder what other messages are being sent on the users behalf (without their
knowing it).
210 Chapter 6
The communicability issue is not necessarily resolved by switching to third-person
discourse. Suppose that the system detects the problem and responds to the origi-
nal sender by saying Your listener is experiencing connection problems. Please,
wait. This is a better phrasing, and the user may even know that the message has
been sent by the system. However, on impulse the user may reply with Thanks
(which would be a cute slip) or with What are the chances that he will be back in
less than three minutes? I dont have much time (which would most probably lead
to a major breakdown). The problem here is that the speaker (the system) is using
a misleading code and medium (the same language and conversational space shared
by humans for natural language communication) to engage in highly constrained,
articially generated communication.
The code and medium problem may be corrected by the use of more appropri-
ate signs (e.g., a visual representation of the listeners state that tells other users that
he is experiencing connection problems). Another important question whose answer
the designers must help the users nd is What if the listener is not receiving the
message? For the sake of efcient communication, the MUApp should help users
make useful decisions about what to do. For example, if the user cant wait long
for the connection to be reestablished, it would be useful to leave a message wrap-
ping up the interrupted conversation and proposing alternatives for what to do next.
If the technology does not offer this kind of recourse in that same conversational
context, the user may feel embarrassed to just leave or feel compelled to wait until
the listener is back. Typically, some email message (referring to the context of their
conversational breakdown) will be sent with an excuse and a proposal for the next
steps. The notion of appropriateness thus extends to the realm of sociability and
politeness, which technology may affect in surprising ways. And it helps one see
that the mere effort of spelling out the contents of the metacommunication message
proposed for MUApps can promote useful reection-in-action during design.
The semiotic engineering of the designers deputys discourse explores very ne
dimensions of human communication. The three conceptual metaphors presented
in section 6.1 emphasize certain aspects of the design vision to detriment of others.
The communication center metaphor usually incorporates attendants, implicit or
explicit agents that can answer the users requests for communication with others.
Attendants can explicitly speak for the designer, not only achieving users goals
through the system, but also talking to them about the system in order to explain
and illustrate features and possibilities, or to help them prevent or recover from
breakdowns. They are the designers deputies, and the greater their communicative
The Semiotic Engineering of Multi-User Computer Applications 211
competence, the greater the chances that users will get the wide spectrum of the
designers message.
In the virtual environment metaphor, the designer must speak through the envi-
ronment itself, or be represented by a special character who inhabits the environ-
ment and may act as a host. If there is no helping agent in the environment, the
complex design vision paraphrased earlier must be fully conveyed through the ele-
ments of the environment, their relations with each other, how they affect each other,
and so on. Since other peoples representations, usually referred to as avatars, are
part of the virtual environment, all interaction with them is radically determined by
interface signs. For instance, in 3-D chats the different positions and orientations
of avatars (e.g., facing in one direction or another) will typically convey important
meanings. When another user is facing in the opposite direction as you are, is he
turning his back on you? If a user/avatar doesnt turn toward an approaching
fellow (by commanding the new orientation of his avatar with a click or keystroke)
what does this mean? How can a user/avatar mean that he is having an interesting
conversation and others are welcome to join in but not to interrupt or disrupt the
conversation? In face-to-face communication, physical signs clearly indicate when
somebody wants or doesnt want to be taken away from a group conversation. Fully
immersive environments, where sensory input is much more rened than in Active
Worlds, may let users manifest their intent with natural bodily signs that have much
higher immediacy than what we experience in the typical semi-immersive avatar sit-
uation. Still, the question is whether appropriate sign categories (with respect to
expression, content, and intent) are available to users and, if not, how they can com-
pensate for the absence.
In chapter 2 we introduced Peirces phenomenological categoriesrstness, sec-
ondness, and thirdness. The virtual environment metaphor for designing MUApps
is an opportunity to use signs related to rstness, namely, the category of qualities
that users can perceive (like other peoples presence or attitude, or certain con-
ditions of the environment). It is also a prime opportunity to explore signs related
to secondness, namely, the category of direct relations that users can perceive (like
the causality between heavy trafc loads or narrow bandwidth and synchronous
communication delays or failures), and thirdness, namely, the category of mediated
(inferential or conventional) relations (like those present in culture, thought, and
language). For instance, if the presence of self and others is represented by avatars,
rstness gives way to secondness and thirdness with respect to representations them-
212 Chapter 6
selves. In face-to-face communication, the presence of somebody is felt before it is
interpreted. In a virtual environment, presence can be felt by signs of secondness
(like the sound and image of the self and others) or thirdness (like the names and
gures of avatars of the self and others). This phenomenon has important conse-
quences for personal relations mediated by computers. Because users are deprived
of rstness, although so many critical nuances of human communication are signs
of rstness, they must resignify rstness as secondness or thirdness, and/or develop
conventions to circumvent the loss of signication dimensions. One of the issues
related to the losses of rstness has to do with trust and truth in chat rooms. For
example, how can a young teenager know that someone she has just met in a chat
room is indeed a friendly fourteen-year-old girl and not somebody else? In other
words, how can we know when another user is being truthful and well-intended or
not? Another is the need to express more explicitly (through secondness or third-
ness) certain attitudes that are clear when rstness can be expressed and interpreted.
For example, if we are not particularly interested in a conversation we are having,
body language may subtly convey our availability to drop off the conversation and
spare us from being rude and saying explicitly I am not particularly interested in
what you are saying. However, if attitude signs must be expressed by caricatured
avatar looks, or explicit availability meters, than we are either necessarily rude to
other people or deprived from experiencing more stimulating communication with
others. Caricatures sometimes invite rude behavior. So, the big challenge for
MUApps designed according to the virtual environment metaphor is that rstness
(and to a considerable extent secondness, too) is nearly impossible to signify appro-
priately. And because such signs play a major role in cultural conventions related
to appropriate interpersonal and public behavior, the quality of computer-mediated
human interaction may be at risk.
Some of the points made here apply to MUApps designed according to the
telecommunications device metaphor as well. When interacting with a virtual fax
machine such as PhoneWorks (in gure 6.5), users also make important judgments
based on signs of rstness, but relative to objects rather than people. The interest-
ing factor in this metaphor is that, unlike the case of virtual environments, the users
activity is ostensively mediated by a toola computer tool, represented as an appar-
ent replica of a physical tool with which the user is familiar. The tool explicitly
creates a distance between the user and the person or people with whom he inter-
acts. The sign of distance is inevitably present. A telephone means (and is meant to
The Semiotic Engineering of Multi-User Computer Applications 213
compensate for) spatial separation in synchronous communication. The role of rst-
ness in such MUApps interfaces is related to the role of perceived affordances
(Norman 1988). The rstness of buttons in gure 6.5 means press me. But rst-
ness may be misleading. The fax page viewed on the fax machine does not behave
exactly like a physical page. For instance, you cannot pull it out of the device and
set it aside on your desktop. The extent of rstness in representing a systems objects
and their behavior is one of the hard challenges in designing direct manipulation
interfaces (Shneiderman 1983; Hutchins, Hollan, and Norman 1986). Direct manip-
ulation deemphasizes the need and value of linguistic discourse, while emphasizing
the value of intuitiveness and affordance. Users are expected to know what to do
just by looking at and touching the artifact. Consequently, the designers deputys
discourse becomes rather limited, because it is a radically reied discoursea dis-
course about ideas and possibilities conveyed strictly through the representation and
behavior of things.
HCI with MUApps designed according to the telecommunications device
metaphor is an interesting opportunity for analysis based on activity theory. The
focus of analysis when this theory is applied to studying HCI is on how human con-
scious behavior is acting through the mediation of various tools and sign systems
(Kaptelinin 1996). In particular, activity theory is interested in how peoples intent
can be affected and shaped by the tools and technologies that they are using to act
in the world. Thus, when using NetMeeting, for example, human communication
is mediated by technology that allows users to employ audio, video, instant mes-
saging, whiteboard, and even shared desktops for synchronous communica-
tion. Human experience is boosted by the technological possibilities opened by co-
interaction with shared applications that are broadcast to the computers of all
users connected in a synchronous communication session.
A semiotic engineering perspective complements activity theory by bringing the
communication of design intent onto the stage of activity, as an explicitly engineered
sign that is available and intentionally transmitted to enrich the spectrum of users
activities. Tool designers become participants in the activity (through the designers
deputy), even if they decline the opportunity to speak overtly about their intent and
prefer to let their product be their spokespersons. Their underlying discourse, as a
motif, permeates and shapes the users experience, hopefully achieving the original
design intent.
Although this metaphor may suggest that sociability issues are farther away than
in the case of the other two metaphors, this isnt really so. In a classroom experi-
214 Chapter 6
ment where two groups of physically dispersed participants were observed while
they used NetMeeting to discuss and decide about the best layout for their new
computer lab, a number of sociability issues came up. Most prominently, users who
had greater difculty managing the whiteboard interface (where layout proposals
could be drawn and edited by all participants) were left out of many important deci-
sions. They were always behind the others in the discussion and could not make
their points in time to get the others attention. This caused a detrimental situation
that, in real collaborative decision-making situations, might bring about undesir-
able effects. Also, since experiments lasted from fteen to twenty minutes, some par-
ticipants grew frustrated and irritable by the end of the experiment, causing the test
to be interrupted before its conclusion.
What the three conceptual metaphors show is that the semiotic challenges of
getting the metacommunication message to users are different in each case. The
communications center metaphor facilitates conversation between the designers
deputy and the users. If the deputy has an explicit representation as an interface
agent (like Ofces assistant, for instance), it may even participate as a legitimate
interlocutor in some kinds of interpersonal or group communication, to help facil-
itate the use of technology. The virtual environment metaphor is semiotically chal-
lenging because the regulating signs for face-to-face human communication, which
designers strive to mimic, fundamentally refer to qualities and perceptions that exist-
ing virtual environments cannot handle appropriately. Such qualities and percep-
tions are evoked or represented by signs of secondness and thirdness that must be
interpreted and learned by users, but cannot grant them the ability to enact the full
spectrum of their social competence. Compensatory and remedial strategies are
usually established by users through what Ellis refers to as social protocols as
opposed to technological protocols (Ellis, Gibbs, and Rein 1991). Finally, the
telecommunications device metaphor preserves the signs of mediation, and physical
separation, in communication, and lets users choose which mediator provides the
best match with their communicative needs and prole. Users dont communicate
with the device, but through it. Hence the idea of transparency (Preece et al. 1994),
of the system receding to the background and throwing the users directly into the
object of their activity (Nardi 1996). The design challenges arise from rstness,
once again, because design intent must be conveyed through qualities that can be
directly perceived and interpreted by users, in a kind of primary semiosis (or intu-
ition) that computer artifacts seldom sustain (being the product of intellectual work
and unique human interpretations).
The Semiotic Engineering of Multi-User Computer Applications 215
The articulation of metaphors with each other is not difcultboth devices and
conversational interlocutors may inhabit virtual environments. Problems are likely
to arise from short-circuiting metaphors, as is the case of anthropomorphized
devices (talking telephones and fax machines) or mechanized communication centers
(where there is no one users can talk to in order to request services, only machines).
These design options will certainly bring about feelings of strangeness and oddity
(which may ultimately be the designers intent, as in games and entertainment appli-
cations) and require that users behave in unnatural ways. An appropriate choice of
metaphors is critically important because MUApp designers determine the types and
quality of communication and action that users will experience through the systems
mediation. Design will determine if they can be cooperative, sociable, fair, and
friendly toward each other. Design will determine if they can have fun, trust, privacy,
and safety while they are together. It will determine the perceptions they have of
each other, and consequently the individual and collective reactions to such per-
ceptions. This shows the importance of an explicit reection on the content of the
metacommunication message in MUApps. The implications of each decision for the
kinds of determinations exerted by design on the users experience should become
clearer as designers think through some of the questions they must answer with
their product.
6.3 Problems and Challenges in Computer-Mediated Group Communication
From the very beginning of HCI studies there have been important calls for
new perspectives on computer technologies for groups. Back in 1986, Winograd
and Flores (1986) characterized computers as a domain for linguistic action
(143), emphasizing the crucial role of human communication as the basis for
coordinated group activity. Their view was opposed to the rationalistic problem-
solving perspective then adopted by most researchers in AI, who pursued the
opportunity to build intelligent decision-support systems to improve the efciency
of organizational processes. Calling the attention of computer systems designers
to the enormous power they have in specifying the range of activities and
values that gain or lose importance in an organization (a point made by others
since the sixties), they proposed that computer technologies should concentrate
on helping people establish and manage networks of commitments through
linguistic action. Such was the gist of LAP, which has gained momentum again in
recent years.
216 Chapter 6
Some years later, Mark Ackerman (2000) proposed that the true intellectual chal-
lenge for CSCW applications was not of a technical order, but rather a social one.
In his view, the most important issues in computer technologies for group action
had to do with knowing more about the users social competence and needs, and
then building technologies that respond more appropriately to these than they now
can. In his terms, this would help us bridge the social-technical gap. At about the
same time, Paul Dourish (2001) also voiced his conviction that more attention
should be paid to what he called social computing. Like Winograd and Flores in
the mid-eighties, Dourish also stressed the social responsibility of systems design-
ers, since even the most isolated and individual interaction with a computer system
is still fundamentally a social activity (56). Design is necessarily interpreted and
used based on social understandings shared by users and designers, and thus
another critical aspect . . . often overlooked is the interaction between the designer
and the user through the system (46).
The semiotic engineering of MUApps focuses precisely on the designer-to-user(s)
communication and aims to help designers understand how their design may or will
shape the interactions of people with tools and with other people (through tools).
As designers elaborate on their intent and on how to convey it to users, they reach
a clearer understanding of how users life and work will be affected by technology.
The common orientation emanating from the contributions of Ackerman, Dourish,
and Winograd and Flores is that designers must carefully reect on the social and
ethical implications of the artifact they are about to produce. The semiotic engi-
neering contribution in this picture is to give form to the materials and meanings
that can trigger such reection.
A walkthrough of the specic contents involved in the designers metacommuni-
cation message can elicit reection on important dimensions of CMC. In order to
support this activity we propose a semiotic inspection of MUApps. The inspection
procedure is guided by four sets of questions related to the metacommunication
message contents. The questions focus on different components of communication,
according to Jakobsons model presented in chapter 2 and briey sketched in gure
6.7. Next to each set of questions are justications and examples of how the inspec-
tion can be carried out (presented as reections).
The Semiotic Engineering of Multi-User Computer Applications 217
I Inspecting interlocutors: Who is speaking? To whom?
Questions:
1 Are the interlocutors represented?
Reection In principle all CMC requires a representation of interlocutors (speaker
and listeners). The only exception might be the implicit representation of self (the
user) in MUApps that follow the telecommunications device metaphor. In this case,
the user may be implicitly represented, as in interaction with any single-user appli-
cation. All action and communication sent is sent-by-self, and all action effects and
communication received are received-by-self. Implicit representations may become
confusing in action and communication logs. For instance, in gure 6.5 there is not
a representation of self.
2 Is the direction of interlocution represented?
Reection The direction of interlocution is crucially important, since it determines
the scope and nature of response. A clear identication of speaker and listener roles
is always required in communication. Implicit representations may be problem-
atic. Being a listener may be implicitly represented by the fact that the user is
receiving/seeing the message. However, for every communication that immediately
218 Chapter 6
Figure 6.7
Focal points for a semiotic inspection of MUApps.
entails action, a clear representation of who is (are) the expected initiator(s) of the
entailed action is necessary for efciency and effectiveness.
Knowing who the other listeners are is important not only for coordination pur-
poses but also for privacy and politeness. Unclear representation and control of the
direction of interlocution may place participants in embarrassing situations. Blind-
copied email messages, for instance, require very careful examination of the direc-
tion of interlocution. A blind-copied interlocutor has a very particular role to play
in communication, and technology usually offers him or her very timid support
against mistaken action. Blind-copied messages are a new breed of complex indi-
rect speech acts introduced by computer technology. In bona de situations, what
is the listener of one such message supposed to do? The message is explicitly directed
to somebody else, but at the same time it is intentionally sent to the blind-copied
listener for a purpose that must not, by denition, be clearly stated in the speakers
message. The darker side of blind-copied messages (and of forwarded messages, to
a considerable extent) is that they somehow facilitate and speed up the effects of
questionable social behavior like gossip and deceit. Of course the technology was
not made to encourage bad manners, but it creates the opportunity for new types
of social conicts that should be considered and discussed at design time.
3 Are representations indexical? Which are the dimensions of contiguity between
representation and referent?
Reection In the context of the representation of interlocutors and direction of
interlocution, indexicality is an important issue. Indexicality is a contiguity relation
(like causality and co-occurrence, for instance) binding representations to their
referent. Genuine indexical signs cannot tell lies about their referent, precisely
because they are not mediated by a third element. The relationship is more direct.
For instance, in MSN Messenger 5.0, if one of your contacts comes online, she
cannot not let you know that she has logged in. This is because there is a causal
relation (that she cannot mediate as she logs in) between the login event and a broad-
cast event that tells all of her contacts that she is online. So indexical representa-
tions of interlocutors and interlocutions give users less control in return for more
delity.
The presence and activity of users in a MUApp is often represented with index-
ical signs. However, we should be aware that computer signs can rarely be indexi-
cal in a genuine sense. Except for signs of hardware activity (e.g., blinking LED
lights, animated cursors, and so on) and very generic causal signs (e.g., response
The Semiotic Engineering of Multi-User Computer Applications 219
time for user and system action), indices of system, process, people, network status,
and the like are all semiotically engineered signs that by convention represent con-
tiguity between what they mean and what they refer to. In other words, a pop-up
window saying Clarisse has logged in is a conventional natural language sentence
(a genuine sign of thirdness) whose presence is contiguous (in this case, causally
related) to Clarisses login in the system. The indexicality is thus established between
the pop-up event (this particular sentence being part of it) and Clarisses current
activity and status.
4 Are representations symbolic? What symbol systems are used? Who produces
them? When and how?
Reection Unlike indices, symbols are mediated representations. Because there is
no contiguity between representation and referent, the relation between them
may be mediated by various levels of semiosis (including inference, action, and
other forms of sense making). As seen in the reection on the importance of
indexicality, we must establish precisely the level at which we want to examine
mediation between representation and referent. In this case, we want to nd out
the kind(s) of mediation that can be used. For example, full-edged linguistic
mediation means that users can (or must) tell what they are doing for others to
know it. By comparison, in a MUApp that adopts this kind of symbolic represen-
tation, online users can only know that Clarisse has logged in if she deliberately
tells them so, or does something else that means she is online (like inviting others
for a chat).
Linguistic mediation can, however, be more constrained and yet very effective.
MSN Messenger itself has useful instances of linguistic mediation regarding users
activity. As soon as she signs in, Clarisse may tell others that she is busy. She may
even tell a lie and appear ofine. This is a gain in control, and a loss in delity,
but it has important sociability implications. Clarisse may want to appear ofine
because she is very busy and knows that one of her colleagues will come online only
to discuss an urgent matter with her. So she wants to be available when that col-
league signs in, but not until then. She prefers to do that instead of saying that she
is busy, which others may read as Im here but please dont talk to me unless
you have something really important to say. The linguistic control that she has in
this MUApp goes as far as the ability to block some of her contacts temporarily,
which amounts to disconnecting the channel for communication with them. With
220 Chapter 6
such tools in hand she can manage her work with greater discretion, making deci-
sions about how not to expose herself and others to socially embarrassing situa-
tions online.
Questions about which signs are available, who can use them, when, and how
should turn the designers attention to the scope and nature of control that users
have upon how they and their respective activities are represented.
5 Are representations iconic? Which are the qualitative dimensions privileged for
iconicity? Are icons static or dynamic?
Reection The importance of iconic signs cannot be underestimated in multi-user
applications. First, as seen in chapter 2, icons are in theory the types of represen-
tations that are most appropriate for bringing out aspects of rstness that are present
in their referent. In other words, icons are efcient ways to represent meaningful
qualities about the referent. Second, as was seen at the beginning of this chapter,
social relations are fundamentally dependent on signicant rstness, like the pres-
ence and attitude of participants during social interaction. But again, computer signs
are all engineered. Thus, iconicity (just like indexicality) must be established at the
right level. Emoticons are perhaps the best example to illustrate how rstness can
be represented in MUApps. The popular smileys, and the ever-growing variations
on the original cartoon-like faces that simulate joy, sadness, surprise, complicity,
and so on, show that they are an efcient sign system (originating in rstness) to
communicate attitude. But, they are a system. In other words, although in many
applications you can make your own emoticon and use it in creative ways, they
follow some important rules and conventions. Graphically, for instance, they may
have to t in particular dimensions. Expressively, they may be (as most emoticons
are) constrained to xed representations of mood and attitude, which is an impor-
tant restriction compared to face-to-face interaction (where mood transitions can
be instantly expressed and sensed by participants).
Another important issue relative to iconicity (and to the previous types of repre-
sentations, as well) is which set of qualitative dimensions in social interaction are
accounted for by representations. Smileys are still a good example. Some attitudi-
nal signs are not prominently conveyed by facial expressions. Relaxed and tense
attitudes, for instance, are often more clearly conveyed by body movement and
posture. The same is true for more or less respectful, more or less empathic atti-
tudes. Therefore, if the domain of a given MUApp is one where these qualitative
The Semiotic Engineering of Multi-User Computer Applications 221
aspects of social relations are important, the designer should strive to let users com-
municate attitude in appropriate ways.
II Inspecting message and code: What is the speaker saying, and using which
code and medium? Are code and medium appropriate for the situation? Are there
alternatives?
Questions:
6 Are speaker and listeners supposed to share the same signication system?
Reection The relevance of sharing the same signication system in communica-
tion is obvious. But this question is intended to go one step further to determine
whether speakers and listeners can use the same signication system they share when
communicating with each other. For example, before installing software or sub-
scribing to online services, users are generally asked to manifest their agreement,
consent, or acceptance with respect to the terms included in licenses and policies.
Although speakers (software manufacturers and service providers, in this case)
and listeners (users) are expected to share the same signication system (a natural
language), they do not share the same expressive systems as far as technological
protocols are concerned. Whereas speakers can say whatever they need to say
in usually long texts, listeners are conned to press one of two or three buttons
saying that they agree, dont agree, or (occasionally) need to know/think more
about it.
The communicative power of speakers and listeners in such situation is very
unbalanced, and not surprisingly users adopt an alienated attitude recognized by a
number of service providers who beg their customers to pay attention to terms and
conditions by saying: Read these carefully. These are not the usual yada yada
yada. Increased efciency and speed (the alleged reason to facilitate the acceptance
of terms in various sorts of formal agreements by allowing users to manifest consent
in one step) is a two-edged sword. Deborah Johnson raises interesting ethical
issues involved in buying and selling software. Although she does not explicitly
address communication (especially communication through software), what she says
is a good illustration of the importance of expressive democracy in licensing and
purchase agreements achieved almost exclusively through interaction with a system:
To force someone to do something is to use the person as a means to your ends
222 Chapter 6
and to disregard their ends. As with honesty, it is easy to understand and avoid
extreme violations, but harder to understand the territory in between (Johnson
2001, 179). Forcing users to say only yes or no to agreements, for instance,
is a way to disregard the users ends. What about questions he or she might
want or need to ask? It is true that users can always call the company, send a fax
or email, and so on. But in the specic communicative context where users are
being asked to decide if they accept the proposed term or not, what if they say
No, meaning Not yet, because I have some questions to ask you or No, because
I dont really understand what you mean. And what if, as is so often the case,
they say Yes, meaning Yes in principle, but I would like to ask you some
other questions or Yes, but I dont really know what you mean or Yes, just
because I trust or hope that you are honest people and would not cause me any
harm if I used your products and services? If the users interlocutor doesnt want
to hear about the actual users intent behind a yes or no what sort of commitment
is there?
One of the most important points of inspecting the extent to which speakers
and listeners share the same signication code, both while interpreting each others
talk and while expressing their own intent, is to nd out whether technology is not
giving unbalanced power to interlocutors and, by so doing, being unethical or
unfair.
7 Can or should the speaker encode or convey his communication differently
(for sake of efciency, effectiveness, cooperation, politeness, privacy, trust, etc.)?
Reection This question is the expected follow-up to the previous one. Does
expressive balance mean that all CMC (including all kinds of user interfaces) should
avoid expressive shortcuts like push-button utterances? No. This is not what it
means. It only reinforces the justication for the main tenet in semiotic engineer-
ing, which is to let the users know the designers rationale and decide what they
want to do.
At the beginning of this chapter, I illustrated a situation where the system sends
a message on behalf of a user who is experiencing connection problems. The idea
of letting a user know that his or her interlocutor is momentarily out of reach is
excellent, of course. The problem is that expressing the message as if it had been
sent by the user in trouble (who may not know about it) creates a problem with
trust. Out of all messages that seem to have been generated by interlocutors, which
The Semiotic Engineering of Multi-User Computer Applications 223
ones actually have? The same example showed that choosing to say Your listener
is experiencing connection problems. Please, wait is better, but still not good
enough. The code used to communicate the problem suggests that the current inter-
locutor (the system) has the same communicative competence as the user and his
former interlocutor. Therefore, it is not surprising if the user engages in conversa-
tion as if the system were able to handle the same spectrum of communication as
any other user. And this will necessarily lead to breakdowns.
Cooperation and efciency must be pursued in communication as well. If there
is no follow-up conversation that the system (in fact, the designers deputy) can
handle after communicating a users disconnection, the code and/or medium for this
communication should be clearly distinct from the code and medium used for inter-
personal synchronous communication. Many applications have efcient ways of
doing this, like combining aural and visual signs that tell someones interlocutor is
out of reach.
But this is not the only situation where mistaken codes and media can interfere
in the success of online communication. In NetMeeting, for example, there are
certain congurations of videoconferencing sessions in which when one particular
interlocutor opens the whiteboard to sketch his idea (even if he has no intention of
sharing it immediately with others) the whiteboard window pops up in every other
users screen. What does the pop-up window mean in this situation? Shall we use
the whiteboard? Have a look at this. Oops!? The problem with NetMeeting
in this situation is that if the user merely wanted to suggest that other users turn to
the whiteboard too, barging into everybody elses screen is not the polite thing to
do. Hopefully, the user will have asked others in the chat or videoconference
window if they agree to use the whiteboard. But in order to do the socially appro-
priate thing, the user should know that the whiteboard is going to pop up in every-
body elses window. Does he know it? If not, the situation can be truthfully
characterized by an Oops! (see communicability utterances in chapter 4), socially
accompanied by an utterance like Im sorry. The problem can be easily avoided
by letting the users know what will happen, and giving them the ability to decide
whether they want to invite others to use the whiteboard (before broadcasting the
process to every other user in communication) or just use it privately for their own
purposes.
In addition to these, design issues also strongly affect the network of commit-
ments that communication typically starts in coordinated action. As Winograd and
Flores (1986) have pointed out, the appropriate response to communication is often
224 Chapter 6
action that evolves over time and affects other people. If speakers and listeners dont
have the appropriate communicative capacities to negotiate meanings before they
turn them into action (or request action), an avalanche of misunderstandings and
failures may follow. Ironically, The Coordinator has been criticized for implement-
ing in technology xed conversational structures as if speech acts could be pre-
dictably mapped onto rigid patterns of action and behavior (Bowers and Churcher
1988). This icon of LAP somehow failed to let users negotiate and control how each
communication should affect others (and the self).
The importance of reecting upon appropriate codes and media for interpersonal
communication online is tied to McLuhans ominous statement ([1964] 2002), The
medium is the message. So, design decisions about how messages are encoded in
a particular medium are decisions about the effects of such messages as they are
sent to their receivers.
8 If there are better alternative codes and media for communication, why isnt
the speaker using them?
Reection When the technology provides alternative codes and media for com-
munication that are more appropriate in the situation than the ones being used,
issues of intentionality can be raised. This is related to the differences captured in
communicability evaluation when I can do otherwise and Thanks, but no,
thanks tags are used to characterize a users behavior. However, in the context
of social relations established in MUApps, this may have important human conse-
quences. If we generalize the enhanced NetMeeting example (assuming that users
have control over whether various devices automatically pop up in everybody elses
screen), sociability in the group may be negatively affected in two very distinct sit-
uations. Suppose that one member of the group knows that he can control auto-
matic pop-ups, and decides to say Thanks, but no, thanks to the designers polite
overture and request that fellow users agree to use the devices rst. Other members
may grow justiably annoyed with this person and segregate him from the group.
But if this person doesnt get the designers message and means to say Shall we use
this device? when he, not knowing better, activates the device in everybody elses
window, he may quite unfairly get the same kind of social response he would have
had he meant to be inconsiderate. And this is obviously unfair. If the designer has
failed to clearly tell this user about the consequences of interaction, then the designer
has exposed the user to social conict.
The Semiotic Engineering of Multi-User Computer Applications 225
This situation calls the designers attention to the effect of default values. These
are normally chosen for protecting or facilitating the users task from a data integrity
point of view. That is, default values suggest the logical nondestructive choice that
users should make in the context of task-oriented activities where objects and
processes dictate the path that best preserve the integrity of their data. But when
objects become subjects, or people, and processes become commitment, default
values must be established according to different scales. Therefore design is very
likely to involve very complex choices between protecting peoples tasks and pro-
tecting peoples relations.
Although choices are very difcult to make, when users know the constraints
imposed by design, they are naturally more forgiving. And design choices themselves
become important signs in group semiosis. If users cannot do otherwise, their inter-
pretation of windows suddenly popping up in their screen, distracting them from
what they were doing, is less likely to generate hostile feelings. It may lead them to
consider that what the pop-ups are showing is potentially important to what they
are trying to do. So, they are motivated to check the message that is, in itself, some-
bodys attempt to collaborate with them. Technology becomes part of the groups
signication system, and it may require that certain social conventions be revised,
replaced, or complemented by protocols that can hold the group together and
sustain sociability in the computer environment.
III Inspecting the channel: Is (are) the listener(s) receiving the message? What if
not?
Questions:
9 Is there a representation of channel and intended receiver(s) status?
Reection The importance of representing context has been extensively empha-
sized in previous questions. Again, an investigation of the nature of signs used in
representations of channel and receiver status is required. Aspects of rstness, sec-
ondness, and thirdness, aligned with timeliness for detecting problems, are impor-
tant, but sociability issues arise here again. In one of the previous examples in this
chapter I mentioned the importance of representing the users status without expos-
ing him or her to embarrassing situations. In a thought-provoking paper about
manners in using contemporary telecommunications technology, Gary Marx (1994)
226 Chapter 6
provides numerous examples of why it has the potential to undermine the dignity
of individuals: In limiting and channeling response possibilities, the format of the
new technologies may alter communication patterns by lessening reciprocal social
skills and weakening the ability to express nuance and complexity (541). Limita-
tions in channeling response possibilities are to a large extent determined by design,
during the semiotic engineering of signs for representing the various dimensions of
social interaction enabled by MUApps. One of Marxs examples, although a decade
old by now, hasnt lost its power:
If [a] caller is left only with the answering machine message, is it rude to hang up without
leaving a message after listening to the recorded message? If so, is it sufcient to simply report
the fact that you called, or should the purpose of the call be revealed as well? (With unpro-
tected caller-ID the question is moot since the time of the call and number of the caller are
automatically recorded. So too with a service called last call return, which permits you to
automatically connect with the last person who called.) In such cases is it presumptuous and
intrusive, or expected and thoughtful, for the person called to return your call on the assump-
tion that you still wish to speak with them? (P. 544)
Notice that if, in a similar situation online, a MUApp designer opts for making
communication more efcient by capturing the identication of everyone who calls
you while you are away from your computer, manners may be at stake in the exact
same way. What do you do if all you know is the name and email of 10 people who
have called you? What do they do if they cant prevent your system from captur-
ing their call, but cant leave a message telling you not to bother to call back? These
issues hinge on others discussed in questions 7 and 8, and naturally lead us to ques-
tions 10 to 13.
10 If the communication channel is defective or the intended receivers are not
available, can breakdowns be prevented or resolved? How?
Reection At the very basic level of communication, this question refers to the
users ability to detect that the intended listener(s) cannot be reached and then decide
what to do. If absence cannot be detected, the speaker may wrongly assume that
his or her message is getting through and take premature follow-up action. What
if the action is difcult or impossible to revoke later? For example, suppose that a
group decides to play bingo together. The play is informal, much like in a face-to-
face social meeting, where one person draws numbers and announces them to the
players, who then check whether the numbers are on their cards. Unlike profes-
sional online bingo sites, where databases randomly generate cards and track the
The Semiotic Engineering of Multi-User Computer Applications 227
numbers on each individually purchased card, in this groups situation the person
who is running the game simply uses a physical bingo card set and randomly assigns
cards to every other group member. The game strongly depends on everybody having
equivalent connection quality. If one of the players has a temporary problem, the
other players and the person running the game should know what is happening and
wait. Otherwise, the disconnected player may be wrongly penalized for technical
problems that he usually cannot control. So, if the player does not get back online
immediately, the group should decide what to do. Unless this group can easily check
the status of communication channels and of other players, the game cannot be
played fairly and, as is often the case with games, some players may get really upset
if they think the others wronged them.
The important point about this question and the example I am using to illustrate
it is not to say that playing bingo online in this way is not a good idea. Rather, we
should examine the possibilities of preventing or circumventing this type of problem,
which is likely to affect other instances of interaction online. The bingo situation
shows that, assuming that this is a group of friends who trust each other and just
want to have some fun together, the group should immediately know if one of the
members experiences technical problems during the game and be able to decide what
to do next. In a social face-to-face situation, if a player receives a phone call, usually
others wait until the call is over to continue the game. And if for some reason the
call is likely to be a long one, the player asks somebody else to check the numbers
for him or her until the call is over. So the question is what the group can do online.
Can somebody check the numbers for the player who is temporarily out? And when
he or she is back, can the group know it immediately? Can somebody report what
has gone on while the player was out? Can the person who is running the game
start logging the announced numbers (which is trivial if the group is using chat to
communicate) and send the log to the absent player as soon as he is back? These
are rather simple things that depend on good design rather than heavyweight tech-
nology. If this game situation is transposed to an organizational environment, where
coordination is needed for action involving liability issues, it is clear that designing
for timely problem detection and, in particular, for taking agile remedial or pre-
ventive action before social consequences grow out of control is a major target.
The bingo example is paradigmatic of a wide spectrum of situations, especially
considering LAP. The solution seems to center on tools that will support the (tem-
porary) logging of group activity and the generation of reported speech (like nar-
ratives) from logged information. All of this could be accompanied by tools that
228 Chapter 6
facilitate catching up with an ongoing process after momentary absences. However,
the typical unstructured chat log is of hardly any use in decision-making discus-
sions. The kinds of tools that will generate the appropriate signs of temporal and
logical relations in discourse are much more elaborate and depend strongly on semi-
otically enriched underlying models of discourse. In them, there are at least two very
important structuring signs that are not present in most chat logs available to date.
One is a useful referential structure saying not only who says what but also who
says what with respect to what. Threaded chats seem to be an interesting alterna-
tive to explore. The other is the rhetorical relation between utterances. An efcient
communication that a certain set of interventions are all explanations of a previous
utterance, or that another set contains interventions that express opposition to the
content of a previous one, helps returning discussants and latecomers gain a quick
view of where and how the discussion is going. So a designer might consider tools
to facilitate such abstract instant views of communication.
IV Inspecting the receivers response and context: Is there recourse if the speaker
realizes the listener(s) misunderstood the message? What is it?
Questions:
This is probably the most crucial set of questions for organizational systems, since
they refer to the scope of control that users have upon the propagation of mistaken
communication and action in computer-supported group activities.
11 Was the listeners response an equivocal action? Can the action be detected
and revoked? How? By whom?
Reection Equivocal action can lead to humorous reaction or to disaster. In
between the two extremes are countless variations of consequences that designers
must consider when building MUApps. Prevention and remedy depend critically on
the timely and correct detection of the mistake. At times, the rst person to detect
it is the one who has taken the action. At other times it is one of the people affected
by the action. And there are also times when only an outsider can detect that some-
thing has gone wrong in a process where action-takers and others affected are so
immersed in the situation that they are blind to their own mistakes.
Technology easily creates specic kinds of blindness that have important ethical
implications. It should not be expected to prevent unethical conduct that cannot be
The Semiotic Engineering of Multi-User Computer Applications 229
prevented in society in general. But it should certainly not facilitate or lead to uneth-
ical conduct, especially by obscuring the consequences of a users action online, or
by not giving him or her enough control to correct a mistake. MUApps are perhaps
the kind of application where designing for error (Lewis and Norman 1986) is
the most critical design imperative. Carefully elaborated metacommunication from
designers to users plays a fundamental role in helping users themselves develop
group practices and strategies that will use the technology wisely and productively.
Decisions about how to halt and revoke mistaken action must be followed by
decisions about who can do it. This may lead to complex conicts of accountabil-
ity. For example, in principle it would be appropriate for the group member who
takes action to be the only one able to change its course or stop it. In a hierarchi-
cal structure, it might be acceptable that other members who have enough author-
ity be able to change and stop mistaken action of subordinate members. But if
technology rigidly implements such hierarchical relations, in some domains the ten-
dency to meet disastrous consequences when mistaken action occurs may be strongly
increased. This may simply be because hierarchical action conicts with timely
action. Maybe the rst person to see the mistake is a group member who is low in
the hierarchy (probably because he is the one most immediately affected), and it
may take a while before the news about the mistake percolates up the hierarchy
to the rst person who has enough authority and availability to take corrective
measures.
The rigidity of rule-based action was one of the major points raised by Winograd
and Flores (1986) two decades ago. Models of group action that are strictly focused
on rationality (where mistakes have little place, if any, and rules take over
opportunity) miss an important share of what may really be going on in human
activity.
12 Was action accompanied by a communication? Can communication about
action be revoked? How? By whom?
Reection This question is an extension of the previous one, and it is purposefully
included as a distinct item for reection because when action is followed by com-
munication, both (action and communication) must be corrected consistently in the
case of mistakes. Saying that the correction has been done before the action itself
is corrected, or saying that it will be done before really knowing if the action can
be corrected, may introduce yet more mistakes and problems in the system.
230 Chapter 6
So pairs of coordinated action and communication deserve even more careful
design.
The interest in examining the relationship between action and communication
(that usually reports the action) resides in the fact that communication may play a
positive and timely role in remedial steps. Communication may traverse hierarchi-
cal structures faster than decisions and actions. And it may help groups manage
critical situations with more agility.
It is important, however, that communication channels and codes support signi-
cation systems where there are distinct signs for communication conveying impor-
tant alerts. They should be differentiated from casual communication, for instance.
Because alerts sent along informal parallel channels of communication can be useful
to control the effects of mistaken action, a careful semiotic engineering of distinc-
tive signs of intent in this case may open useful opportunities for handling break-
downs online.
13 Does revoking the communication entail a new network of commitments?
Should the network be facilitated by the system, and, if so, to what extent?
Reection The purpose of this question is to call the designers attention to the
fact that the use of technology for correcting mistakes made through (or because
of) technology itself may be problematic. In previous questions I discussed the poten-
tial value of providing informal means of communication in technology that can
primarily control work processes more rigidly (e.g., workow systems). However,
as Dourish (2001) points out, users appropriate technology in surprising ways.
Although design can substantially determine the range of possible appropriations,
from the moment that collective semiosis takes place there are literally innite pos-
sibilities that fall within this range.
In chapter 5 I discussed the notion of semiotic innities and showed that such
innities may be considerably different from each other. MUApps are an interest-
ing case where decisions must be made about which innities are more important
than others and why. Workow systems introduce typed innities that enable the
automatic generation of various signs (including reports, newsletters, alerts, and
even work processes and commitments). But these innities are monotonous (in
every sense of the word). They are predictable, uniform, hopefully unfailing,
and for this very reason potentially alienating. So the availability of informal
communication channels that will serve as safety nets in case of major breakdowns
The Semiotic Engineering of Multi-User Computer Applications 231
may end up absorbing some important portions of a workow system simply
because group members are more comfortable and satised when they are directly
in touch with each other. And when this happens, the workow system itself is likely
to begin telling lies.
For example, suppose that in the workow system there is a process requiring
that a particular report be sent from User A to User B by a certain date. The report
is expected to be sent through the system, an action that automatically generates
various signs. First, the system reports that User A has done his task and that its
now User Bs responsibility to make the next step in the process. Second, it informs
the upper management that the state of the process is owing as expected and that
questions and requests about it should be sent to User B. But, what if User A sends
the report to User B using the informal channel, andeven betterpredicting the
consequences of this decision, he also informs the upper management, through the
same informal channels, that he has done so. In this situation, everybody but the
system knows what they have to know. The groups understanding of the activity
state is correct, but the systems understanding and representation of it is incorrect.
This makes the system lose credibility and defeats the whole purpose of using work-
ow technology.
Design decisions about preventing and recovering from mistaken action and com-
munication are very hard to make, and there is no recipe for successful designs. The
ethical step to make is to let users know and decide as much as possible about
design, both during the design process and later when the MUApp is in use. This
naturally leads to customization and extension issues (see chapter 5), associated with
the notion of system accountability (Dourish 1997). The use of efcient and effec-
tive self-representations of computer systems may attenuate the disparities between
the technical and the social perspectives in CSCW. And this is at the very heart of
semiotic engineering. The self-representation of the system can be explicitly elab-
orated in the designers deputys discourse. This allows designers and users to be in
touch and to share the responsibility for the use of technology, coming closer to
meeting a variety of ethical imperatives.
The semiotic inspection of MUApps can be a powerful design and evaluation tool.
A systematic walkthrough of its questions can draw a professionals attention to the
tight relations among various elements of design. Part of this inspection has been
presented elsewhere in the form of a communicative adequacy test for online com-
munities technology (de Souza and Preece 2004).
232 Chapter 6
6.4 Pragmatic Principles, Speech acts, and Technological Protocols
As computer applications began to be used for social, emotional, cultural, and even
political purposes, it became clear that HCI designers needed to know much more
about the social sciences and humanities than they used to. At the same time, it was
also clear that the complexity of issues involved in designing good MUApps requires
knowledge that cannot be captured in teach-yourself formulas. Complexity cannot
be reduced by following a series of steps or checking a number of properties if the
background knowledge that gives meaning to procedures and checklists is not there.
So the aim of semiotic engineering is only to help designers organize the design
space, formulate relevant questions, search for answers (through research, experi-
mentation, consulting with other experts, professional training, etc.) and develop
their skills to communicate solutions in such a way that they can fairly share the
responsibility for their design with those who will buy it and/or use it. Semiotic
engineering provides reading, an interpretation of complex problems, and promotes
a kind of reection that is typically conducive to interdisciplinary regimes in research
and professional practice. In this reading is the seed for contents that belong to
the metacommunication message and for the communicative competence that
the designers deputy must have to support productive and pleasurable user
experiences.
In this perspective, the role of specialized knowledge generated in the context of
other disciplines is to inform design and promote reection. Pragmatic principles,
for example, can be used in the design of MUApps. If the cooperation maxims pro-
posed by Grice (1975), or even the politeness maxims proposed by Leech (1983),
are implemented in a MUApp, they are encoded in the applications signication
system. In other words, they have an impact on the designers deputys behavior, on
the interpretation that it generates for users behavior, and on the types of commu-
nication (and their corresponding effect) that users can have with the deputy and
with other users. If the Quantity Maxim is encoded in the MUApp signication
system, for instance, then certain consequences may follow. The designers deputys
discourse may have little or no redundancy when communicating with users, since
redundancy (by denition) multiplies the occurrence of the same piece of informa-
tion in discourse. Similarly, structured user-system and user-user communication is
likely to contain little or no redundancies. This design decision has pros and cons,
as some of the examples in this chapter have already suggested. The pros center
around efciency and speed in communication, whereas the cons center around
The Semiotic Engineering of Multi-User Computer Applications 233
difculties in detecting mistakes. Redundancy is an important factor in communi-
cation, since it can help verify content validity continuously and speed correction
if, in a particular context, content is found to be wrong.
The inuence of the Quantity Maxim in design can reach far. For example, email
addresses of online community members are critical for community managers. This
information must always be up-to-date in order to ensure that members can be
reached. In order to ensure that email addresses are up-to-date, MUApps designers
may choose, among other possibilities, to have the system periodically ask every-
one if they have changed their email address, to use banners to warn users about
the importance of keeping their email addresses up-to-date, or to use the users email
address as his or her login. The rst and second choice may be a nuisance for people
who are not likely to change their email addresses. The two types of communica-
tion violate the Quantity Maxim for these users, since not only do they already
know that it is important to keep their email addresses up-to-date (and dont need
to be told about this all the time), but they are not likely to change their addresses
to begin with. This makes the third choice seem very attractive, because users will
realize that they must notify the community manager of a change of email address
the next time they login using the old address. However, the effect of this strategy
is that the users identication is his or her email address. The designer must there-
fore be careful to reveal this identication to the community manager only. Other-
wise, all email addresses of community members will be known to everyone. And
this has negative privacy implications that designers want to avoid. As a result, users
would probably end up with two identicationsone for the community manager,
and one for community members (e.g., like a nickname). This example calls atten-
tion to causal relations between representation and referent and to indexical signs.
A minimalist interpretation of the Quantity Maxim in design may augment the
designers tendency to choose indexical signs in user-system and user-user commu-
nication. Indexical signs may trigger very powerful semiosis (especially if the causal
relations binding representation and referent are part of long causal chains that can
be traced in the users activities). But precisely because they can do it, they may
expose more about the users than they are willing to expose.
A similar reasoning can be applied to how speech acts are encoded as part of sig-
nication systems designed for MUApps. The Coordinator is a good example of the
possible problems generated by causal encodings of the primary structure of speech
acts. Promises, or commissive speech acts, generate commitments. However, phras-
ing an utterance as a promise doesnt necessarily make it a promise. The listener,
234 Chapter 6
for example, must take it as a promise. And a listener can always decide not to take
a promise as a promise. Besides, there are different degrees of commitments involved
in promisesat times a promise is not kept and there are no consequences (social,
psychological, or other) for any of the involved parties (promisor or promisee). At
other times, quite contrarily, an utterance is barely phrased as a promise, and yet
the consequences of not keeping it are disastrous. The nuances of meaning in each
case are so subtle that even promisor and promisee may be mistaken about the
strength of the commitments entailed by promises. Thus, it is not surprising that a
causal encoding of speech acts into conversational structures that will track that
promises are kept, that directives are met, and so on eventually brings about more
tension or misunderstandings than expected.
Ellis, Gibbs, and Rein (1991) distinguish between technological and social pro-
tocols. Technological protocols are group interpersonal contact practices that are
built into hardware or software (i.e., encoded into the MUApp signication system),
whereas social protocols are mutually agreed-upon practices that the group formally
or informally decides to follow when they are together online. The authors note
that when design privileges social protocols over technological ones, processes
become typically more adaptive, but this choice may also lead to distraction and
inefciency in group activities. On the other hand, technological protocols can be
overly restrictive, preventing the group from developing their own idiosyncratic way
of interacting with each other. Since designers of MUApps must eventually choose
when to go for one over another kind of protocol, it is helpful to help them
represent their vision and rationale, to let the signs present in design semiosis
materialize somehow and talk back to designers during reection in action.
Prates (1998; Prates and de Souza 1998) and Barbosa (2002) have developed a
conceptual architecture model for MUApps design-support tools called MArq-G*.
The model has been conceived to help designers materialize part of their vision and
rationale about group communication processes. The interesting thing about MArq-
G* is that it encodes certain communicative dimensions that designers must think
about during design. As decisions about the values and signs associated with each
of these dimensions are made and expressed in a design language, heuristic rules
about the interaction among such values and signs can be used by the interpreter
of MArq-G*, which then prompts the designer to think about potential inconsis-
tencies and trade-offs. The MArq-G* architecture is presented in gure 6.8.
Although Prates and Barbosa initially conceived of MArq-G* for only a subset
of the dimensions discussed in section 6.3, the value of the model is that it
The Semiotic Engineering of Multi-User Computer Applications 235
accommodates expansions to account for as many dimensions as designers wish to
consider at design time. For the sake of illustration, let us look at an instantiation
of a specic set of dimensions and rules used by Barbosa to illustrate the types of
reections that MArq-G* can support during design. The instantiation refers to a
description of NetMeeting and focuses on the way this MUApp works when the
whiteboard is opened by one of the group members who are together online. In
order to clarify the nature and consequence of decisions, let us recall how Net-
Meeting works in the context of a hypothetical scenario. It involves three partici-
pantsA, B, and Cwho have different hierarchical roles in the social context
where they are. A is B and Cs superior. So these three are using the chat tool to talk
about a conference A and B have been to, and C doesnt quite understand the phys-
ical layout of the exhibit area. So A and B have different ideas of how to tell C about
it. Whereas B chooses to write a more detailed description of the area, A decides to
make a drawing on the whiteboard and explain the spatial conguration with it.
What happens is that as soon as A opens the whiteboard, it simultaneously pops up
on B and Cs screens. B interrupts his typing in the chat tool and momentarily
wonders how the whiteboard appeared on his screen. A is not completely aware of
236 Chapter 6
Figure 6.8
MArq-G* components.
the kinds of disruption she may be causing to B or C, because in her mind she is
only drafting the exhibit area, which she will be talking about when she nishes.
Barbosas illustration focuses on the illocutionary force taken up by certain speech
acts when they are encoded in the technological protocol. The relevant portion of
the design description, which we have translated and adapted from the original in
Portuguese (Barbosa 2002) to t the purposes of this chapter, is as follows:
Roles Supervisor is a role; Helper is a role.
Group Members A is a member; B is a member; C is a member.
Classes of Objects Tool is a class of objects.
Objects Chat is an object; Whiteboard is an object.
Role Attribution A is a supervisor; B is a helper; C is a helper.
Group Hierarchy Supervisors are superior to helpers.
Shared Objects Chat is shared by A, B, and C; Whiteboard is shared
by A, B, and C.
Communicative Prole Any member can communicate directive speech
acts to other members regarding objects that they share.
Now, in the heuristic interpretation rules of MArq-G* there is knowledge about
the illocutionary force of directive speech acts. The illocutionary force is the com-
bination of the speakers intent and the contextual conditions that must be true so
that the speech act can be accomplished (e.g., sincerity conditions, preparatory con-
ditions). Two of these rules are:
Rule #N In a hierarchical group structure, directive speech acts
sent from superiors to subordinates have force of order.
If order is not the intended illocutionary force, there
must be explicit signs to distinguish and verify illocu-
tionary forces that can be associated to directive speech
acts in this case.
Rule #N + 1 In a hierarchical group structure, directive speech
acts sent from subordinates to superiors, or communi-
cated among members at the same hierarchical level,
have force of suggestion. If suggestion is not the
intended illocutionary force, there must be explicit
signs to distinguish and verify illocutionary forces
that can be associated to directive speech acts in
this case.
As the MArq-G* interpreter processes the descriptions for this group of three
people, it detects that there are different illocutionary forces for directive speech
The Semiotic Engineering of Multi-User Computer Applications 237
acts: orders and suggestions. The MUApps signication system may explicitly
encode these two meanings, by having differentiated lexical signs for one and the
other, or ascribe these meanings to the signs occurring in the interlocutors semiosic
processes. Explicit markings give users much more control over illocutionary force.
For example, if there are no markings, all directives sent from member A (see pre-
vious example) to B and C will be always interpreted as an order. But there may be
times when A just wants to make a suggestionnot force her ideas on others just
because she happens to be the others superior. These reections are triggered by
the contents of rules N and N + 1.
Prates and Barbosa have implemented a design language and a knowledge base
with nearly sixty heuristic rules written in Prolog. Their original design tool cuts
across most of the questions presented in section 6.3, but only a few types of reec-
tions have been included in the knowledge base. Nevertheless, the type of tool they
have developed can be expanded to include not only a larger portion of the reec-
tions in section 6.3 but, in fact, additional questions and their corresponding reec-
tions. An important aspect of this tool is that the character of the rules in the
knowledge base is merely descriptive. Building on the same NetMeeting example,
and assuming that there are no signs in the MUApps signication system to differ-
entiate between illocutionary forces (as is indeed the case in NetMeeting), an order
and a suggestion from A to B and C are indistinguishably encoded as launching the
tool in all the subordinate participants desktops. However, subordinate participants
may minimize (and ignore) the directive to use the tool, and just keep doing what-
ever they were doing before. This design feature in NetMeeting introduces an ambi-
guity that touches on questions 11, 12, and 13 in section 6.3: What if the listener
misunderstands communication? The problem with NetMeeting is that the misun-
derstanding may take a while to be detected. How will the supervisor know if sub-
ordinates have minimized the whiteboard on their respective screens? NetMeetings
designers have decided to assign all these decisions to the social protocol. The tech-
nological protocol does not help participants control the status of shared applica-
tions to this level of detail. Thus the knowledge base may warn designers that certain
breakdowns may follow from this decision. But it will not prevent designers from
proceeding with their design decisions and overriding the rules relative to detecting
misunderstandings.
The particular implementation of MArq-G* made by Prates and Barbosa is less
important than the type of computer tool that can be developed to help designers
238 Chapter 6
express their rationale and reect upon the potential consequences of their
decisions. A valuable by-product of using such a tool is that nal design decisions
and trade-offs may be encoded in the designers deputys discourse about the
technology and help users have a clear idea of the MUApps they are using for
interacting with others online. It is particularly important for designers to share
responsibilities with the users of the technology they produce, especially in the
context of MUApps that can be used for social, emotional, cultural, and political
purposes.
6.5 Technological Signs and Their Impact on Group Semiosis
Ellis, Gibbs, and Reins early discussion (1991) about decisions on what to assign
to social protocols and what to assign to technological protocols when talking about
computer applications for groups might lead designers to suspect that social proto-
cols may always override, modify, or control what is encoded in the technological
protocol. For example, if NetMeetings technological protocol appears rude to other
group members, for barging onto their desktops with a new tool, the right thing to
do it to negotiate the launching with other members through chat or video-
conference. This strategy may work, although it leads to the sorts of consideration
about manners that Gary Marx (1994) makes. Technological protocols, however,
reach much farther than this.
In a study about the expectations that potential MUApps users have about this
kind of technology (da Silva et al. 2003), we have found that a group of people
who met regularly face-to-face for a weekly academic seminar had the following
categories of expectations:
Regarding the presentation of the content of their discussions online, they expected
that they would have
information ltering;
summaries and abstracts;
structuring and visualization of information.
Regarding the ability to reproduce or approximate the kinds of interaction that they
had face-to-face, they expected that they would have
enriched representation and control of the context of communication;
the ability to use gestures and to express their feelings and attitudes.
The Semiotic Engineering of Multi-User Computer Applications 239
Regarding privacy issues, they expected that they would have
privacy among members, in particular:
the ability to have private conversations,
the ability to organize individual and collective spaces;
privacy regarding visitors and new members:
the ability to decide which portions of group discussions and materials visitors
would be able to see.
Regarding the style and exercise of leadership, they expected that they would have
encouragement to participate in group activities;
mutually agreed procedures for decision making;
coordination or moderation of discussion sessions.
Regarding their participation in group decisions, they expected that they would have
the ability to make group decisions about allowing or not allowing visitors to access
group discussions;
the ability to make group decisions about what to do with members who do not
contribute to discussions;
the ability to make group decisions about how to present collective information (to
insiders and outsiders).
A semiotic inspection of three lightweight contemporary environments popular
among online groups showed that what such potential users expect and what they
get from such environments is considerably far apart in a number of cases. Part of
the distance is due to questionable design decisions that, for example, prevent users
from organizing their discussions and materials in effective ways, or make it dif-
cult for them to control how much of their activities outsiders can have access to
when they are visitors to the group. Another part of the distance is due to the tech-
nological restrictions of lightweight environments. The signs that express feelings
and attitudes, for example, are not particularly rened in such environments. Body
language and facial expressions are typically sacriced in nonimmersive MUApps,
causing nonverbal communication to suffer considerably from lack of expression.
All the discussion about signs of rstness in section 6.3 is applicable in this case.
However, the most interesting indication of this study has been the possibility that
the rule-based nature of computer programs (which all MUApps ultimately are, no
matter what the share of decisions that designers assign to social protocols emerg-
ing from interaction in the computer medium) is incompatible with the dynamics
and exibility of human interaction.
240 Chapter 6
In part, this was the point made by Winograd and Flores (1986), when they called
to the attention of organizational systems designers the fact that all computer pro-
grams necessarily affected the relationships and the achievements of people who
used them. But it seems that the authors new foundations for design, replacing
goal-directed rational models of human activity with models of human coordina-
tion for action through communication, still dictates and imposes important con-
straints on social interaction. Our study suggested that the symbol-processing nature
of computation requires that every group online must be represented and compu-
tationally interpreted and controlled according to a xed set of rules. It is as if all
groups, even the most ephemeral and informal ones, necessarily had to obey a
certain number of unchanging rules. These basic rules determine the necessary con-
ditions for the group to be created, for somebody to become a member, for any
member to express whatever they want to express, for any other member to react
to a members expression, and so on. Unlike with natural groups, whose members
typically have an unconscious knowledge (often a mere intuitive feeling) of the prin-
ciples that hold the group together and give meaning to their activities, with online
groups self-consciousness and observation of rules is an imperative. Thus, by
necessity, an extensive share of human social competence is transformed online. The
place and role of intuitions is shifted to other spheres of human interaction, and a
number of familiar signs in face-to-face communication must be resignied within
a technologically constrained semiotic space.
By examining the expectations of participants in our study, we concluded that the
difculty in designing MUApps is not only that there is so much to know about and
that decisions are always so hard to make. It is also that the current model of com-
putation that determines the implementations of computer applications for groups
does not allow us to meet some cherished social values like spontaneity, informal-
ity, exibility, and freedom. These signs were explicitly mentioned in the answers
given by participants to questions asked in our studys interviews.
Paul Dourish (2001) has recently raised a related point. He writes, HCI,
from its very beginning, took on the trappings of the traditional computational
model and set out its account of the world in terms of plans, procedures, tasks, and
goals (4). His critique is directed to the role that representations have had in this
tradition. Representations have led to disembodiment in actionthat is, to a
separation between mind and body, between thought and action. And Dourishs
point is precisely to reconcile the two, and to achieve embodied interaction, inter-
action rooted in the ways in which people (and technologies) participate in the
The Semiotic Engineering of Multi-User Computer Applications 241
world (189). However, the author concedes that software is a representational
medium.
If representations are inevitably at the heart of software artifacts, then again it
seems that although the kinds of rules processed by MUApps can be made more
exible, more adaptive, more embodied, the semiotic engineering of representations
will always determine the types of rules that users will have to know and live with
as they use computers to interact with other people and achieve a wide variety of
goals and effects.
6.6 Epistemic Tools for the Design of Multi-User Computer Applications
In this chapter I have introduced epistemic tools that can help designers formulate
and explore semiotic issues involved in the design of MUApps. First, I presented
three basic conceptual metaphors that are used in isolation or in combination with
each other in practically all contemporary MUApps: the communication center
metaphor, the virtual environment metaphor, and the telecommunications device
metaphor. I have shown that each metaphor motivates the use of different types of
signs, and by so doing each favors certain aspects of communication to the
detriment of others. I have also shown that these metaphors can be articulated with
each other in coherent ways, but the discourse of the designers deputy can only be
explored to the degree of nesse introduced in previous chapters of this book if a
linguistic system can be extensively used. Strictly visual signication systems are
excessively conning to communicate the whole spectrum of meanings involved in
group interaction online through the mediation of artifacts resulting from human
interpretation and imagination.
Then I characterized the content of the metacommunication message from
MUApps designers to MUApps users. From this characterization I derived a
semiotic inspection procedure consisting of a set of questions about the structure
of user-system and user-user communication. For every question I presented types
of reection that can be triggered by such attempts to nd an appropriate answer.
These types referred to channels and codes of communication, to the categories of
signs in encoded signication systems, to the values (social and other) that provide
a backdrop for interpretation of computer-mediated communication, to the possi-
bilities of detecting and correcting communicative mistakes, to the roles and expec-
tations held by interlocutors, and so on.
242 Chapter 6
By aligning the concept of technological protocols to encoded signication
systems in MUApps, I have analyzed the trade-offs of design when it comes to imple-
menting meanings that are part of causal relations expected to hold when they are
used in technological contexts. This led me to examine the fundamental effects of
computation on group relations and activities. Computer programs are rule-based
symbol processes that challenge many users aspirations with respect to the nature
and quality of their interaction online, such as spontaneity, informality, exibility,
and freedom. Computer programs require that users follow the rules encoded in
them, something that naturally promotes a higher degree of self-consciousness and
regulation than in many natural face-to-face social relations.
Although I have not and cannot give denitive answers to the issues raised in this
chapter, the message repeatedly stated for designers is to improve the communica-
tion of their design vision to users. This will not only allow them to share more
fairly with the users the responsibility for the technology they have designed, but it
will also trigger the users collective semiosis in potentially surprising directions and
clever practices.
The Semiotic Engineering of Multi-User Computer Applications 243
III
Questions
Use your light,
but dim your brightness.
Lao Tzu
7
Reection
I am now coming to the end of this book, when it would be time to present my
conclusions. However, in the semiotic vein so strongly emphasized in the previous
chapters, I deliberately choose to talk about reections rather than conclusions. The
book is a testimony to my own ongoing semiosis, and I hope that the signs I have
used to express my perceptions will stimulate the readers semiosis along a new path
of exploration and discovery. There will be counterevidence to many of my abduc-
tions, new signs for continuing research, all very welcome as in any scientic
endeavor.
The following sections are three notes meant to sound merely as a soft nal chord.
The rst evokes the relationship between semiotics and computation; the second,
the relationship between semiotics and design; and the third, the role of epistemic
tools for HCI designers and users. Some signs on the road ahead resonate with this
chord, and with them I nish this book.
7.1 Semiotics and Computation
There is a short story by Umberto Eco called On Truth: A Fiction (Eco 1988),
which represents through ctional narrative a longstanding philosophical debate
about the denition of meaning and where it resides. The short story is worth men-
tioning in this nal reection, because it touches deep on meaning and accounta-
bility, two issues seldom discussed in HCI despite the relevance of their implications.
The plot of the story is an insightful allegory. Sometime in the future, the members
of an outer-space expedition all die on another planet, defeated by dysentery. Not
knowing if this incident calls for retaliation, the leaders of the project on Earth
decide to nd out rst what has really happened. A second expedition is then sent
to the planet in order to investigate whether the members of the previous one have
been poisoned by the natives. It is known that they drank as water what the natives
call water. This expedition tests the natives to nd out which feelings or mental
representations they have when they hear the word water. But the problem is that
natives dont speak like humans. When a child approaches a hot stove, for instance,
a mother says: Oh my God, he will stimulate his C-bers! (Eco 1988, 41).
Because this expedition fails to advance the investigation, a third expedition is sent
to the planet. The mission is now to nd out if, in the expeditioners view, the natives
can understand the meaning of any sentence at all. The method they use is to explore
the meanings stored in this planets computers. Why? Because computers necessar-
ily have representations of whatever it is that meaning means to the people who
build the programs that run on them. AI programs, in particular, have the addi-
tional capacity to make meaningful computations on representations. Therefore,
their algorithms are themselves representations of what meaningful computations
mean to the people who build them.
The story develops in a kind of reversed Turing test, where one of the expedition
members steps into the carcass of an empty computer and talks to another com-
puter as if he were a computer himself. As the human performs wonderfully in the
reversed Turing test, the computer reveals all it knows about the representations
and meanings it can compute upon. The computer lets the expedition member know
that it denes itself as only a semiotic machine, inexorably interpreting expres-
sions in accordance with higher-order expressions. But it cannot, however, explain
why or how its own interpretation process works. All the computer succeeds in
saying about its masters is I dont know if my memory is the same as that of my
masters. According to my information, they are very uncertain about what they have
inside them. . . . That is the reason why they set me up. They know what I have
inside me, and when I speak in a way that they understand, they presume that they
have the same software inside them (Eco 1988, 59).
This allegory touches on philosophical issues that permeate everything we do in
HCI. Positivist traditions have encouraged the view that computer users have in
mind some achieved, unchanging, objective knowledge, purposes and perceptions
that can be modeled and mirrored in various types of software artifacts. So there
have been calls for technology that supports more embodiment, situatedness,
and deeper social and context awareness, all of which are fundamentally right
in denouncing reductionist attempts to squeeze the whole spectrum of users
understanding and experience into mechanical systems that follow strict logical
procedures.
248 Chapter 7
However, one of the hard problems in HCI and in computer science in general is
precisely what program producers and program consumers take program represen-
tations to mean. As suggested in Ecos ction, when they understand program rep-
resentations they tend to presume that such representations are somehow true.
But they are only true to what programs have inside. Why are program produc-
ers and consumers inclined to see truth in programs? And what does this truth
mean?
In our daily lives, we are constantly exposed to things that are not true. Many of
them are just plainly stated as suchthey are formulated as conjectures, hopes,
fears, jokes, possibilities, assumptions, ideas. And we are fully capable of dealing
with things that are explicitly not true, or not even meant to be true. The hypoth-
esis advanced in this book is that, to date, the main foundational theories in HCI
have tacitly assumed that computer programs must contain true statements about
the users insideabout their mental behavior, their intent to act in the world,
their way to perceive and relate to the environment, and so on. No theory has overtly
questioned who is making such statements, or how far these statements can be
carried when it comes to designing and engineering computer artifacts. Psy-
chophysical laws, probabilistic principles, statistical evidence, and even minute
observations of past and gone experiences have been often expected to spell out the
truths to be modeled and encoded in computer programs. But the disorienting evi-
dence of how such truths are not true in so many contexts of use has been taken
as an indication that there is something wrong with the theories.
But there isnt necessarily anything wrong with the theories. What may be wrong
is how we interpret them, and what we expect to be able to do with them. Semi-
otic engineering makes an important switch in this perspective, by claiming explic-
itly that computer programs represent what a designer takes to be true about the
nature, the usefulness, and the impact of the artifact being designed. It frames soft-
ware as a very particular kind of discourse, communicating ideas elaborated by
people, to people, and about peoplenot ultimate truths. A designer should thus
be radically honest, at design time and at interaction time, replacing the notion of
truth with that of beliefs. The goal of HCI in semiotic engineering is to allow users
to react to technology just as they react to communication that makes them an offer.
What if they used the technology being offered? Do they want to use it? How do
they want to use it? Would they wish to adapt it, to make changes? Users are nat-
urally more inclined to relate to technology that wears its weaknesses and limita-
tions on its sleeves, but much less inclined to be tolerant with technology that is
Reection 249
built and expressed as a claim (rather than a belief) about who they are and what
they want.
Theories, in my perspective, should not be used as the providers of truths and
denitive answers to be encoded in computer programs. And this applies to semi-
otic engineering itself, of course. Most of the hard problems in HCI will never have
a nal answer. Designers must just learn to deal with them and share their efforts
and responsibilities with users. Talk to them and let them talk back to them, at
interaction time, right there where the action is. This is not the kind of embodi-
ment that Dourish (2001) has proposed pursuing, but it is a kind of semiotic embod-
iment, which leaves traces of the designers sense making in the artifacts interface
and allows users to relate explicitly to the designers mediated rational presence in
the artifacts that we build.
As Ecos short story shows, computers are a privileged medium for the deepest
possible inspections of what is involved in communication because, unlike humans,
they algorithmically produce all and only the meanings that have been explicitly
selected to be part of the signication systems encoded in them. The relationship of
computer meanings with each other can be inspected to exhaustion. A program is
the full specication of how representations affect and are affected by others. A
program is a unique sign of its creators thought, a sign that tells a lot more about
its origin, purpose, and behavior than any other sign in our culture. But, typically,
neither program producers nor program consumers take programs to be such signs.
Semiotic engineering is a theory oriented toward enabling designers to explore this
possibility and to see which consequences follow from it.
7.2 Semiotics and Design
Semiotics and design have very strong relations. They have been previously explored
with specic reference to visual design in HCI (Marcus 1992; Mullet and Sano
1995), to computer systems design in general (Nadin 1997), and also to architec-
tural design (Broadbent 1980; Medway and Clark 2003) and design in principle
(Newton 2004). In this book I have used Schns reection-in-action perspective
on design (1983) as a background theme for presenting and discussing semiotic
engineering.
The technical rationality perspective (Simon 1981) that Schn has so strongly
criticized in the eighties covers a considerable portion of design in engineering. It
supports designers judgments about the constraints imposed upon artifacts by
250 Chapter 7
established laws, principles, norms and, values. The key in the technical rationality
perspective is the word established. Once there is a xed set of assumptions that
cannot be questioned or changed by designers (for scientic or other reasons), a
number of consequences follow from them. Enduring assumptions of this sort even-
tually constrain the portions of design affected by them in a limited number of
problem-framing and problem-solving possibilities that become and are passed on
as technical knowledge.
Technical knowledge carries certain self-accomplished prophecies in it. In civil
engineering, for instance, the design of HVAC (heating, ventilation, and air condi-
tioning) systems incorporates technical knowledge that can guide the patterns for
distributing heat loads. The professional belief in such technical knowledge is so
strong that AI systems have been built to nd solutions and provide explanations
for HVAC design (e.g., Garcia and de Souza 1997). This kind of sophisticated tech-
nological tool, praised by many in engineering, even allows designers to assign
unpredicted values to certain design parameters. By so doing, HVAC designers intro-
duce modications in the systems knowledge base, strictly speaking, novelties,
renements, or improvement of previous technical knowledge. However, as
I have shown in preceding chapters (especially in chapter 5), the system can
indeed handle innite possibilities, but they are all innities of the same kind. Many
expert systems demonstrate the value placed on technical knowledge by certain
professions where design is involved. But not much attention is paid to the self-
perpetuating beliefs that are passed on and tacitly accepted as true practical
knowledge just because they can be expressed in computer-processible form and
can generate representations of solutions to representations of practical professional
problems.
As Winograd and Flores (1986) have shown, the assumption that software engi-
neering can or should follow the same paradigm as other engineering disciplines is
fundamentally awed because the stuff of computer artifacts is language (representa-
tions), not matter. In other words, whereas physics predicts that certain heat loads will
cause materials and uids to behave in particular ways, cognitive psychology, social
sciences, linguistics, or semiotics can not usually predict that certain representations
will cause people to behave in this or that way. Nevertheless, computer science can,
and does, predict that computer programs cause representations to generate and
derive other representations. As I have suggested in section 7.1, the producers (includ-
ing designers) of computer artifacts have sometimes taken computer programs for car-
riers of universal truths about users. And they have used pieces of knowledge from
Reection 251
the humanities and social sciences for the same purposes that civil engineers pick
knowledge from natural sciences, for example. Even HCI researchers have often mir-
rored themselves in this paradigm, using ontologies and methodologies that have
worked for research in natural sciences as if they would by necessity work in the same
way (if at all) for research in human and social aspects involved in software engi-
neering. Results have not often been encouraging, of course, and doubts about the
value of theoretical contributions of HCI as a eld have been voiced (Landauer 1991;
Rogers, Bannon, and Button 1994; Shneiderman et al. 2002).
Schns reection-in-action perspective is liberating in many ways. First, it rescues
the freshness and creativity of our intuitive views about design from the conning
territory of black art, where rationalistic views tend to place it. By dening design
problems as unique, Schn is also saying that every solution is unique. And the
uniqueness of design problems and solutions stems from the designers inescapable
freedom to interpret and frame the opportunity and purpose of design in totally
contingent ways. Second, it recognizes the role of professional training and skills in
design and places them at the same level as other professions generally taken to be
more technical. The kind of professional knowledge that must be taught to design-
ers is nevertheless a very particular one. Schn qualies it as epistemological knowl-
edge, that is, knowledge that will help designers come to know the nature of unique
problems and decide how they can best be solved. Solutions may occasionally be a
matter of selecting one or more problem-solving techniques that have worked in the
past, but most of all they may also and more often be a matter of invention and
experimentation, just as in scientic research. Hence the critical role of epistemic
tools for design, and the need to equip professionals with the ability to construct
design problems, use design research methods, hypothesize design solutions, decide
how much they are worth, and express them as a design product. And third, Schns
perspective subtly changes the role of artifacts consumers. Whereas consumers
relate to concrete artifacts typically produced by traditional engineering disciplines
as utilities or appliances (words that are particularly revealing in this context),
when design begins to be viewed as intrinsically innovative the consumers reaction
must itself be innovative and judicious.
The representational nature of computer programs suggests that HCI design is a
semiotic endeavor (Nadin 1988). But expecting that a direct application of semi-
otic methods (most of which are essentially analytic) will lead to the solution for
preconceived classes of HCI design problems throws us right into the pitfall that
Schn is talking about. Semiotic engineering is not meant to be a predictive or
252 Chapter 7
explanatory theory in the sense alluded to by Shneiderman et al. (2002). It is neither
a theory that will give quantitative estimates of human performance with input
devices (e. g., Fitts Law), menu traversal, or visual scanning, nor one that will
guide future designs, such as Don Normans seven stages, Clarks common ground,
or . . . direct manipulation (688). Predictions about human performance and
informed design guidelines may, however, be the result of a designers reection
while using some of the epistemic tools presented in this book.
Some may hope that it will be a kind of generative theory, in Shneiderman et al.s
sense (2002), assuming that it will open up new knowledge domains and guide us
to innovative applications (688). But this is not my own vision of semiotic engi-
neering, if the kind of guidance expected is generative in a technical sense. Gener-
ation, in mathematical and linguistic domains for example, is a rule-based causal
process by which new symbols are derived from root symbols (like theorems from
axioms, sentences from a vocabulary, and so on). This is exactly the antithesis of
this theorys goals, since it is totally incompatible with Schns perspective on design,
which I have chosen to adopt.
However, a loose interpretation of generative theories as trigger theories, theories
that will initiate associative (semiosic) processes, rather than derivational (rule-
based) processes, might allow me to say that semiotic engineering is in fact a likely
generative theory of HCI. The epistemic tools presented in this book should trigger
the designers semiosis, which will eventually produce signs that will be encoded in
a computer applications signication system. Then these signs will become part of
the users semiosis, both when interpreting the designers message and when pro-
ducing their own and unique interactive discourse. So the theory binds design signs
and user experience signs in a continuum that hopefully will spark designers and
users imagination and open some paths for innovation.
7.3 Epistemic Support for Designers and Users
Semiotic engineering, like other semiotic accounts of HCI, views computers as a
medium for communicative exchanges. It is a unique medium in that interlocutors
do not all belong to the same phenomenological category. Through programs, users
communicate with other users, designers communicate with users, and programs
themselves communicate with other programs. In other words, programs are also
media, even though they are primarily messages, representations of meanings pro-
duced by designers and programmers.
Reection 253
A reversed formulation of McLuhans principleas the message is the medium
achieves more than a rhetorical effect. It suggests that the purpose of meanings (the
message) encoded in computer programs is to support the exchange of other mean-
ings (to be a medium). This book has shown that the semiotic engineering inter-
pretation of what Reeves and Nass (1996) call the media equation (italics, mine)
is slanted toward this particular reading of the relationship between message and
medium. The aim of the theory is to support designthat is, to help designers get
their message across to users. It is not an explanatory theory of how users make
sense of computer artifacts. Semiotics, for example, would be an appropriate theory
for this end. The reversed formulation of McLuhans principle opens the avenue for
designing different media in the computer medium. It is a more detailed articula-
tion of the semiotic possibilities enabled by computers, one in which the role and
purpose of design stand out more obviously.
Once the message in the computer medium carries the design intention, it is
important that both designers and users get support for achieving metacommuni-
cation. The success of HCI design can be measured not only by the users complete
acceptance of the designers message as such, but also by the users interventions to
adapt and repurpose the message for unanticipated goals and contexts. The contact
between designers and users is established through the metacommunication
message. Consequently, all the epistemic tools presented in this book center around
the construction of this message. In chapter 4 I explored various types of break-
downs that may happen in communication between users and the designers deputy.
These breakdowns offer insights into the construction of online help systems, where
the designers deputys discourse can be more elaborately presented to users. In
chapter 5 I explored the conditions for encoding the metacommunication message
into computable signication systems, and the limits of customizations and exten-
sions with which users can adapt and repurpose this message. Chapter 6 presented
a more sophisticated exploration of the metacommunication message as medium,
identifying the particular contents in it that can affect the mutual relations and per-
ceptions of users who are brought together within the domains of multi-user appli-
cations. The metacommunication paraphrase also broadly characterizes the contents
that users are expected to detect in the designers deputys discourse. Such contents
are important epistemic sources for the users sense-making processes. As users
deepen and rene their interpretations about the possibilities of the medium, their
own semiosis is impregnated by the explicit signs expressed by designers in the
designers deputys discourse.
254 Chapter 7
Communicating knowledge is the central focus of semiotic engineering. HCI
design produces artifacts that must come to be known through communication.
Certain signs occurring in the designers semiosis at design time will crystallize as
signs encoded in the artifacts signication system. And this signication system is
the same through which and with which users will express whatever their semiosis
produces. And no one can predict the exact shape and length of users semiosic
paths.
In semiotic engineering designers gure as participants in (and not just builders
of) the users experience. They may choose to appear in disguise, letting the object
of design speak for them, or to tell users explicitly what their vision and intent is.
The choice of how they will participate in HCI is itself a semiotic choice. The theory
aims at investigating computer-mediated and computer-encoded semiosis, in an
attempt to raise both the designers and the users awareness of the medium created
by the metacommunication message. Verbal communication is of course a prime
means for expressing the variety of representations contained in computer programs.
And the more users know about such variety, the greater the scope of possibilities
for them to innovate and evolve. But verbal communication may step back and
make way for other types of communication, whose limitations may be an impor-
tant part of the designers message. In computer games, for example, the designers
goal (and the users delight) may be precisely to exercise imagination and play
according to highly constraining rules that stimulate and challenge ones intellect
and emotions. So designers can themselves play with signs and experience novel
forms of communication, novel codes, novel channels, novel messages, and novel
contexts.
The intended effect of a semiotic engineering of HCI is that computer
artifacts themselves will be like epistemic tools for their users. If through interac-
tion users gain new knowledge about their world (not only about the artifact!),
and if this knowledge gives them the same level of satisfaction and achievement
envisioned by designers, the engineered product is successful. The most important
metrics of semiotic engineering success is not a measure of how much of the
designers meanings coincide with the users, but rather a measure of how
much of the designers meanings are used in successful expressions of what
users themselves wanted to do. This metric can give us useful insights about the
intended versus the perceived value of technology, rather than about the difculty
versus the ease of performing a xed range of tasks with it, and the time it takes to
learn it.
Reection 255
One of the potential results of emphasizing epistemological aspects in design, and
of viewing design products as epistemological products themselves, is that users will
know more about designers. Interestingly, all the user-centered mantras in HCI have
repeated the need to know the users. This is undeniably a fundamental requirement
for designing good products. But we have never seen a suggestion that users should
know designers. The dominant metaphor that software artifacts are like concrete
artifacts (like mechanical tools, electric appliances, pieces of furniture) has promoted
the view that we dont need to know about who designed them and for what. But,
as I have tried to show in this book, software artifacts have an intellectual nature.
All about them is a representation, a selected expression of selected meanings for
selected purposes. And users must share such representations with designers in order
to express what they want to do, even if what they want to do does not coincide
with what designers thought they would want to do. Isolating users from the origin
and motivation of representations they have to use while interacting with software
is a major constraint, since it creates the need for representations to be self-evident,
which is precisely what we have the greatest difculty achieving in any sort of com-
munication. If designers assume that their representations are self-evident to users,
and users fail to understand them and use them productively, very negative feelings
may arise. Designers may be frustrated by their incapacity to produce a self-evident
artifact, and users may be frustrated by their incapacity to understand a self-evident
artifact. But once the myth of self-evidence is brought to light and we begin to see
software artifacts as just another type of communication, nonevident meanings
naturally become the topic of clarication dialogs, and users and designers are not
trapped in a situation where every doubt, mistake, and deviation is a reason for
blame and frustration. Perhaps by viewing software in this new communicative light,
we can identify the epistemic value of information technology and make more
intrepid explorations of where all of this can lead us.
7.4 Signs on the Road Ahead
John Carroll (2003) has said that the scientic fragmentation in the eld of HCI is
one of the important barriers to be transposed if we want to make progress. Because
there are too many approaches, too many techniques, too many technologies, too
many theories, seeing the big picture is extremely difcult. As a consequence, when
building interactive software artifacts we are not always sure of (and maybe
not even aware of) where and why old or new knowledge applies. This situation
decreases our ability to make appropriate and justiable choices in HCI design, and
256 Chapter 7
increases the feeling expressed by somethat scientic theories cant help us build
quality artifacts any better than sheer individual talent and professional practice. It
also jeopardizes the value of crucially important scientic ndings in HCI, because
of equivocal expectations about the scope of their account.
Scientic fragmentation can be improved by an appropriate epistemology. But to
date, because of its interdisciplinary character, the epistemology of HCIif explic-
itly addressed at allhas itself been a patchwork of epistemological borrowings
from psychology, computer science, anthropological disciplines, engineering, and
design. The result of this patchworked epistemology is that sorting scientic issues
from nonscientic ones, distinguishing between essence and circumstance, and
even identifying the permanent goals of the discipline have been extremely difcult.
Implemented systems are at times taken for sufcient representation of the discipli-
nary knowledge they claim to contain. Fundamental research questions are aban-
doned because technological evolution has replaced the infrastructure where they
were being investigated. Premature implementations of incomplete and imprecise
knowledge are marketed and stir economic resources to accelerate progress based
on assumptions that are not clearly understood by researchers themselves.
Our motivation to work on semiotic engineering has always been the elasticity
of the theory in accommodating within the same ontological perspective bits and
pieces of HCI knowledge that cant be handled satisfactorily by any other single
theory. Previous attempts to use a single theoretical framework to bridge
phenomena in the human and the computer domain, like the human information
processor (Card, Moran, and Newell 1983), have been the result of carrying a
model that works for one into the domains of the other, where it originally does
not belong. They have been metaphorical accounts. The semiotic base of the theory
presented in this book, however, extends over human and computer domains
without having to resort to metaphors. Computers do not have to be viewed as
humans, or humans as computers, for the ontology or methodology of the theory
to be consistent. Humans and computers are native citizens in the domain of semi-
otics, and hence of semiotic engineering. And this is a peculiar and advantageous
situation held by this theory in comparison to many others in this eld. It means,
for example, that the epistemology of semiotics is in principle amenable to sup-
porting a very wide range of scientic investigations in HCI, and it carries the
promise of ghting the battle of fragmentation, so sharply expressed by Carroll
(2003).
But this theory may also be felt as a threat. First and foremost, it embraces the
view that human meanings cannot be fully known, and consequently not be
Reection 257
completely (or adequately) encoded in computer systems. This is threatening because
it shakes the foundations of many design and evaluation methods and practices,
where the ultimate match between system meaning and user meaning determines
the goal of design and the quality of software. If the users meaning is an evolving
unlimited process of interpretation, halted and resumed for contingent reasons, and
shaped by unpredictable factors that emerge during the process itself, the aura of
rationality that HCI has so strongly struggled to preserve seems to vanish in the air.
In defense of semiotic engineering, I can invoke two main arguments. One is that
taking users meanings to be xed and ultimate targets that can be captured in com-
puter systems doesnt change the reality of what users do. And this reality is often
disconcerting for any one who takes time to sit and watch what happens when users
interact with software. The other is that humans are all perfectly equipped to handle
unlimited semiosis. We do this all the time. So, maybe the trouble with computers
is not really that they cant capture and account for all that we mean while we inter-
act with software, but rather that this is not what they should be doing in the rst
place. Maybe computer systems are more like storytellers than looking glasses. Every
story has an author, and listeners naturally know this. They may identify themselves
with one or more characters, or perhaps with none of them, and still think that the
story is a great one. The same is true of systems. Even if users dont recognize them-
selves as the model user in the system, they may step into the model users shoes
and enjoy the plot just as much.
Another reason why semiotic engineering may be felt as a threat is that it may
be wrongly interpreted as a denial of all technological developments achieved in
recent years, a regression into verbose and tutorial styles of interface. Talking about
the designers deputys discourse and communicating design rationale to users
sounds like a case for pervasive AI and verbal communication as the exclusive
alternative in HCI. But, although verbal communication allows for ne types of
expression, not easily achieved by other alternatives, there are numerous kinds of
signication systems that can be encoded in the computer medium. They can be
explored by semiotic theories and combined with one another in order to maximize
the efciency of communicative processes involved in HCI. Abundant verbal
communication is neither a requirement nor a necessarily good alternative for inter-
action. The challenge is to spot the best opportunities for efcient nonverbal com-
munication, when nonverbal signs are powerful enough to communicate more
clearly and better the designers understanding and intent, and to provide access to
verbal communication whenever we spot the opportunity (or need) to negotiate
258 Chapter 7
potentially incongruous meanings. In other words, verbal communication should be
an available resource, not the only possibility for interaction.
Semiotic engineering may also be mistaken as a self-defeating theoretical alter-
native, because it rests on the assumption that users are interested in knowing more
about the artifacts that they use. Since people dont usually care to read online
documentation and prefer trial and error to using help resources that come with
applications, seeking to let them know more about the system they interact with
may seem like an illogical choice. The immediate and supercial argument in favor
of the theory is that, if we look at human communicative behavior, the economy of
conversation in goal-oriented situations suggests that interlocutors do go into very
sophisticated meaning-negotiating processes under certain basic conditions. For
example, they may do so when they believe that getting the right meaning is needed
for the achievement of their goals; they believe that the negotiation will not cost
them much; and they believe that the benets of negotiation outweigh the risks of
misunderstandings. Trial and error suppresses meaning-negotiation processes and
replaces it with repeated attempts at guessing rather then asking. But this strategy
may turn into a problem when guesses fail one after the other, or when guesses dont
reach far enough (e.g., when only part of the meaning is understood, and the part
that is not understood leads to other communicative breakdowns in situations that
are not perceived to be related). For this reason semiotic engineering uses commu-
nicative breakdowns during HCI as an opportunity to convey useful elements of the
design vision and intent. This may substantially increase the impact of this kind of
knowledge on the users semiosis and raise the users awareness about the ontolog-
ical nature of software artifacts. But my deeper argument in favor of the theory is
that semiotic engineering questions an implicit ethics in HCI, according to which
users denitively dont need to know more about computers. They just need to be
able to use them. The theory shows that, as a rule, designers and developers have
been building computer artifacts that are not meant to be known. And thus it is not
surprising that people are not inclined to, or not even capable of, knowing them.
Users have had little ability to use the technology judiciously, and have instead been
using it in good faith. From the moment we realize the arbitrariness of so many
choices we make, we become ethically responsible for externalizing such choices,
just like food manufacturers are ethically responsible for letting consumers know
the ingredients they add to their products.
This book, like many others, asks more questions than it gives answers. More-
over, the spectrum of questions is very wide, and at times very deep. My purpose
Reection 259
is not to immobilize research and professional practice with the amazing com-
plexity of issues involved in designing interactive computer artifacts. The message
of semiotic engineering is not that designers should not build these artifacts before
they have the answer to all the questions that have been asked in this book. On the
contrary, the message is that they might begin to create very different patterns of
HCI right away if they explicitly present the computer systems they design as intel-
lectual artifacts. By so doing, they will let users understand them, use them, relate
to them, value them, and transform them, just as they do with other intellectual
artifacts. The road ahead, however, is a very long one, for culture is not changed
by individual speculation and will. There must be many more signs that semiotic
engineering is in fact a useful theory of HCI. Some of the more immediate ones
may come from exploring issues such as:
What does software intentionally produced as a metacommunication artifact
look like?
Most software produced to date has been designed and evaluated with an empha-
sis on facilitating operation and control. It is very difcult to nd interfaces designed
to tell users about the various use strategies supported by the system, and when
or why one strategy can be better than the other. As the Acrobat example has shown
(see chapter 1), this kind of information is usually buried in online documentation
and is really not designed to be communicated effectively to users. Exploring ways
to communicate tactical and strategic aspects of software use, directly through the
interface, should yield designs that are very different from what we now see.
How do users perceive this type of software, in comparison with software that is
not so produced?
A whole range of comparative studies taking software that is traditionally designed
in accordance with user-centeredness and usability guidelines, and software designed
as a metacommunication artifact, can teach us many lessons. They are crucial to
advancing knowledge in semiotic engineering and can contribute substantially
to HCI in general. Such comparative studies have been difcult to carry out so far
because there havent been many different theories to support the elaboration of
comparable hypotheses and methods in HCI design and evaluation.
How can the designers intent be expressed during the design process? Which
portions of it can make a difference in the users experience if users are explicitly
told about them? Which difference? Why?
260 Chapter 7
Not all aspects of design intent are equally relevant for a user to know. Some are
too technical, others are plainly commercial, and still others may not really make
any positive difference in the quality of a users experience. On the contrary, if
designers begin to talk about such uninteresting issues, interaction may indeed get
worse. So a key topic for research is to identify which aspects of the whole design
vision can improve the users experience. Capturing and conveying them appropri-
ately can help us investigate the real effect of this type of knowledge on the users
experience and begin to formulate explanations about it.
In how many different ways can users be told about the designers vision? How
does this kind of communication impact the users attitude toward computer
technology? What may this impact mean for the software industry? What may it
mean for individuals and society?
As this book has illustrated in numerous instances, communication is as varied as
we can imagine. Thus, there are unlimited possibilities to convey the designers
message to users. Finding effective ways to do it will enable us to let users know
more about computers and computing. In the long run, if we are successful
with this sort of communication, we will probably begin to see changes in the users
attitude toward computer technology, and we can investigate the new kinds of
intervention that technology can make in individual and collective life.
One of the most important signs of what semiotic engineering really means will
come from how research itself is conducted. Because this is an integrative theory
that gives us instruments not only to ght the scientic fragmentation identied by
Carroll but also to position ourselves more actively and responsibly in view of how
users experience software, perhaps it will inspire a greater number of epistemolog-
ically informed research projects. It will be interesting to see whether the new kinds
of issues and results so achieved bring about new understanding in our eld and
give us, researchers and practitioners, a genuine sense of improvement.
Reection 261
Notes
1. In fact, designers may specify the encoding of meanings in an interactive application, but
software developers program this specication into the actual computer system that users
interact with. The passage from specication into programming may of course introduce mis-
takes and unspecied arbitrary programming decisions that directly affect the users experi-
ence. This passage is itself a semiotic process, which can be analyzed with semiotic tools, but
we will assume an ideal situation, where the designers specication if faithfully translated
into a computer program. However even in non-ideal situations where programming
introduces novel or mistaken meanings into the application, compared to its specications,
the fact that designers and users are united under the same communicative process holds true.
2. Saying that the designers deputy is the system is itself the result of collapsing the
medium with the message.
3. The classical Turing machine model used as a formal characterization of all effective pro-
cedures presents some important features that can distinguish it from an adequate model of
human activity based on meaning interpretation. The machine operates with a nite vocab-
ulary (i.e., input tape symbols). Input symbols that do not belong to the machines vocabu-
lary and for which, consequently, there are no specied state transitions to be made, cause
the computation to halt and fail. On the one hand, human mental machinery does not
halt in this way in the presence of unknown vocabulary items. For example, if one doesnt
not know what metalinguistic means in a sentence like The use of metalinguistic resources
helps users nd what interface signs may mean, one can try to infer this meaning from
context and nd a reasonable solution (e.g., metalinguistic means something related to
the explanation or denition of what signs mean). This is an effect of ongoing semiosic
processes and abductive reasoning that characterize human meaning interpretation. On the
other hand, when communicating ideas, one can invent new words, on demand, to ll an
instant expressive need. By denition, these new words are not part of the interlocutors
vocabulary but are expected to be interpreted appropriately by force of contextual, etymo-
logical, grammatical, or other factors. For instance, if one says something like This device
is so tiny that it would be more appropriate to refer to it as a ngertop rather than a
palmtop, the non-English word ngertop typically poses no difculty for ones interlocu-
tor in communication. A classical Turing machine would not produce such inventive output
unless there were rules to generate it (but then what would be the invention?). Moreover,
unlike the human speaker, who can explain what he means by ngertop and why he used
this invented word, the machine cannot say why it uses any word. The reasons and
explanations for a given machines behavior can only be given by a higher-order machine.
This constraint is associated to the halting problem in computation. Turing machines
can take other Turing machines as input. But if they take themselves as input there may be
input symbols for which the machine will never stop transitioning among its internal states,
producing from zero to an innity of results.
Although an innite production of results might tempt us to say that such machines are
appropriate models of unlimited semiosis, they are not. Unlike what happens in human unlim-
ited semiosis, the machine cannot decide that it is satised with what an input symbol means.
Some higher-order machine must provide it with the criteria for satisfaction and give it a
command (input) to proceed with computation later, if there is the need, the opportunity, the
resourcesin short, the reason why it should continue.
4. The Turing machine model for computer programs indicates that some programs can
compute forever, producing from zero to an innite number of output symbols (all the same,
all different from one another, with periodic repetitions, with random repetitionsall these
are possibilities). Human unlimited semiosis is different from computation that produces an
innite sequence of symbols because, as mentioned in section 4.4, human unlimited semio-
sis is halted and resumed because the human interpreter decides to do it. Computer programs,
however, cannot make decisions about their own computation. They can only execute them.
Decisions about their execution have to be made by another program that contains the meta-
language of the program being executed.
264 Notes
Bibliographic References
Ackerman, M. 2000. The intellectual challenge of CSCW: The gap between social require-
ments and technical feasibility. Human-Computer Interaction 15(2): 181203.
Alexander, C. 1977. A Pattern Language. New York: Oxford University Press.
Andersen, P. B. 1997. A Theory of Computer Semiotics, 2nd ed. Cambridge: Cambridge Uni-
versity Press.
Andersen, P. B. 2001. What semiotics can and cannot do for HCI. Knowledge-Based Systems
14(8): 419424.
Andersen, P. B., B. Holmqvist, and J. F. Jensen. 1993. The Computer as Medium. Cambridge:
Cambridge University Press.
Aristotle. 1980. Aristotles Categories and Propositions (De interpretatione). Trans. with
commentaries and glossary by Hippocrates G. Apostle. Categoriae. Grinnell, IA: Peripatetic
Press.
Aristotle. 1984. The Rhetoric and the Poetics of Aristotle. New York: The Modern Library
College Edition.
Austin, J. L. 1962. How to Do Things with Words. Cambridge, MA: Harvard University
Press.
Barbosa, C. M. A. 2002. MetaCom-G*: Especicao da comunicao entre membros de um
grupo. M.Sc. diss. Departamento de Informtica, PUC-Rio, Rio de Janeiro.
Barbosa, S. D. J. 1999. Programao via interfaces. Ph.D. thesis, Departamento de Infor-
mtica, PUC-Rio, Rio de Janeiro.
Barbosa, S. D. J., and M. G. de Paula. 2003. Designing and evaluating interaction as con-
versation: A modeling language based on semiotic engineering. In 10th International Work-
shop on Design, Specication and Verication of Interactive Systems, DSV-IS03, 1327.
Funchal, Ilha da Madeira.
Barbosa, S. D. J., and C. S. de Souza. 2000. Expanding software through metaphors and
metonymies. In International Conference on Intelligent User Interfaces IUI2000, 1320. New
York: ACM Press.
Barbosa, S. D. J., and C. S. de Souza. 2001. Extending software through metaphors and
metonymies. Knowledge-Based Systems 14(12): 1527.
Barnard, P., J. May, D. Duke, and D. Duce. 2000. Systems, interactions, and macrotheory.
ACM Transactions on Computer-Human Interaction 7(2): 222262.
Bastien, J. M. C., D. L. Scapin, and C. Leulier, 1999. The ergonomic criteria and the ISO/DIS
9241-10 dialogue principles: A pilot comparison in an evaluation task. Interacting with Com-
puters 11(2): 299322.
Bellotti, V., S. Buckingham Shum, A. MacLean, and N. Hammond. 1995. Multidisciplinary
modelling in HCI design . . . in theory and in practice. In Proceedings of ACM CHI 95:
Human Factors in Computing Systems, 146153. New York: ACM Press.
Bdker, S. 1989. A human activity approach to user interfaces. Human-Computer Interac-
tion 4(1): 171195.
Bowers, J., and J. Churcher. 1988. Local and global structuring of computer mediated com-
munication: Developing linguistic perspectives on CSCW in Cosmos. In Proceedings of the
1988 ACM Conference on Computer-Supported Cooperative Work, 125139. New York:
ACM Press.
Braa, K., and R. Vidgen. 1997. An information systems research framework for the organi-
zational laboratory. In Computers and Design in Context, ed. M. Kyng and L. Mathiassen,
381400. Cambridge, MA: The MIT Press.
Broadbent, G. 1980. Architectural objects and their design as a subject for semiotic studies.
Design Studies 1(4): 207216.
Brown, G. 1995. Speakers, Listeners, and Communication. Cambridge: Cambridge Univer-
sity Press.
Brown, J. S., and P. Duguid. 1992. Enacting design for the workplace. In Usability: Turning
Technologies into Tools, ed. P. S. Adler, and T. A. Winograd, 164197. Oxford: Oxford Uni-
versity Press.
Card, S. K., T. P. Moran, and A. Newell. 1983. The Psychology of Human-Computer Inter-
action. Hillsdale, NJ: Lawrence Erlbaum.
Carroll, J. M. 1998. Minimalism beyond the Nurnberg Funnel. Cambridge, MA: The MIT
Press.
Carroll, J. M. 2003. HCI Models, Theories, and Frameworks. San Francisco: Morgan
Kaufmann.
Carroll, J. M., and M. B. Rosson. 1987. The paradox of the active user. In Interfacing
Thought: Cognitive Aspects of Human-Computer Interaction, ed. J. M. Carroll, 2628.
Cambridge, MA: The MIT Press.
Cawsey, A. J. 1993. Explanation and Interaction: The Computer Generation of Explanatory
Dialogues. Cambridge, MA: The MIT Press.
Chandler, D. 2002. Semiotics: The Basics. New York: Routledge.
Clark, H. H. 1996. Using Language. Cambridge: Cambridge University Press.
Cunha, C. K. V. 2001. Um modelo semitico dos processos de comunicao relacionados
atividade de extenso aplicao por usurios nais. Ph.D. thesis, Departamento de Infor-
mtica, PUC-Rio, Rio de Janeiro.
Cypher, A. 1993. Watch What I Do. Cambridge, MA: The MIT Press.
266 Bibliographic References
Cypher, A., and D. Caneld-Smith. 1995. KidSim: End user programming of simulations. In
Proceedings of ACM CHI 95: Human Factors in Computing Systems, 2734. New York:
ACM Press.
da Silva, S. R. P. 2001. Um modelo semitico para programao por usurios nais. Ph.D.
thesis, Departamento de Informtica, PUC-Rio, Rio de Janeiro.
da Silva, E. J., C. S. de Souza, R. O. Prates, and A. M. Nicolaci-da-Costa. 2003. What they
want and what they get: A study of light-weight technologies for online communities. In Pro-
ceedings of the Latin American Conference on Human-Computer Interaction, 135146. Rio
de Janeiro: PUC-Rio.
Danesi, M., and P. Perron. 1999. Analyzing Cultures. Bloomington: Indiana University Press.
de Paula, M. G. 2003. Projeto da interao humano-computador baseado em modelos fun-
damentados na engenharia semitica. M.Sc. diss., Departamento de Informtica, PUC-Rio,
Rio de Janeiro.
de Saussure, F. 1972. Cours de Linguistique Gnrale. Paris: Payot.
de Souza, C. S. 1993. The semiotic engineering of user interface languages. International
Journal of Man-Machine Studies 39(4): 753773.
de Souza, C. S., and J. Preece. 2004. A framework for analyzing and understanding online
communities. Interacting with Computers 16(3): 579610.
de Souza, C. S., and K. Sedig. 2001. Semiotic considerations on direct concept manipulation
as a distinct interface style for Learnware. In Anais do IHC2001-IV Workshop sobre Fatores
Humanos em Sistemas Computacionais, 229241. Porto Alegre, RS: SBC.
de Souza, C. S., S. D. J. Barbosa, and S. R. P. da Silva. 2001. Semiotic engineering principles
for evaluating end-user programming environments. Interacting with Computers 13(4):
467495.
de Souza, C. S., R. O. Prates, and S. D. J. Barbosa. 2003. Adopting information technology
as a rst step in design: Lessons learned from working with Brazilian social volunteers. Inter-
actions 10(2): 7279.
de Souza, C. S., R. O. Prates, and T. Carey. 2000. Missing and declining affordances: Are
these appropriate concepts? Journal of the Brazilian Computer Society 1(7): 2634.
Delio, M. 2001. Eudora retards ames. Wired News. Available at
http://www.wired.com/news/technology/0,1282,38723,00.html. Accessed Sept. 12, 2000.
DiGiano, C., and M. Eisenberg. 1995. Self-disclosing design tools: A gentle introduction to
end-user programming. In Proceedings of DIS95, 189197. New York: ACM Press.
Dourish, P. 1997. Accounting for system behavior: Representation, reection, and resource-
ful action. In Computers and Design in Context, ed. M. Kyng and L. Mathiassen, 145170.
Cambridge, MA. The MIT Press.
Dourish, P. 2001. Where the Action Is. Cambridge, MA: The MIT Press.
Dourish, P., J. Holmes, A. MacLean, P. Marqvardsen, and A. Zbyslaw. 1996. Freeow: Medi-
ating between representation and action in workow systems. In Proceedings of the 1996
ACM Conference on Computer-Supported Cooperative Work, 190198. New York: ACM
Press.
Bibliographic References 267
Eco, U. 1976. A Theory of Semiotics. Bloomington: Indiana University Press.
Eco, U. 1984. Semiotics and the Philosophy of Language. Bloomington: Indiana University
Press.
Eco, U. 1987. Travels in Hyperreality. London: Picador.
Eco, U. 1988. On truth: A ction. In Meaning and Mental Representations, ed. U. Eco, M.
Santambrogio, and P. Violi, 4159. Bloomington: Indiana University Press.
Eco, U., M. Santambrogio, and P. Violi. 1988. Meaning and Mental Representations.
Bloomington, IN: Indiana University Press.
Ellis, C., S. Gibbs, and G. Rein. 1991. Groupware: Some issues and experiences. Communi-
cations of the ACM 34(1): 3958.
Farkas, D. K. 1998. Layering as a safety net for minimalist documentation. In Minimalism
beyond the Nurnberg Funnel, ed. J. M. Carroll, 247274. Cambridge, MA: The MIT
Press.
Fetzer, J. H. 1988. Program verication: The very idea. Communications of the ACM 31(9):
10491063.
Fischer, G. 1998. Beyond Couch potatoes: From consumers to designers. In Proceedings
of the 3rd Asia Pacic Computer Human Interaction Conference, 29. Piscataway, NJ: IEEE
Computer Society.
Fitt, P. M. 1954. The information capacity of the human motor system in controlling ampli-
tude of movement. Journal of Experimental Psychology 47: 381391.
Flores, F., M. Graves, B. Harteld, and T. Winograd. 1988. Computer systems and the
design of organizational interaction. ACM Transactions on Information Systems 6(2):
153172.
Garcia, A. C. B., and C. S. de Souza. 1997. ADD+: Including rhetorical structures in active
documents. AIEDAMArticial Intelligence for Engineering, Design and Manufacturing
11(2): 109124.
Gelernter, J., and S. Jagganathan. 1990. Programming Linguistics. Cambridge, MA: The MIT
Press.
Gibson, J. J. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifin.
Golightly, D., and D. Gilmore. 1997. Breaking the rules of direct manipulation. In IFIP TC13
International Conference on Human-Computer Interaction, ed. S. Howard, J. Hammond,
and G. Lindgard, 156163. London: Chapman Hall.
Grice, H. P. 1975. Logic and conversation. In Syntax and Semantics, Speech Acts, vol. 3, ed.
P. Cole and J. L. Morgan, 4158. New York: Academic Press.
Gruber, T. R. 1993. Toward principles for the design of ontologies used for knowledge
sharing. International Journal of Human-Computer Studies 43(56): 907928.
Gugerty, L. 1993. The use of analytical models in human-computer interaction. International
Journal of Man-Machine Studies 38(4): 625660.
Halasz, F. G., and T. P. Moran. 1982. Analogy considered harmful. In Proceedings of the
Conference on Human Factors in Computing Systems, 383386. New York: ACM Press.
268 Bibliographic References
Hayakawa, S. I. 1972. Language in Thought and Action. Orlando, FL: Harcourt.
Hollan, J., E. Hutchins, and D. Kirsh. 2000. Distributed cognition: Toward a new founda-
tion for human-computer interaction research. ACM Transactions on Computer-Human
Interaction 7(2): 174196.
Holst, S. J. 1996. Directing learner attention with manipulation styles. In Proceedings of CHI
96: Conference Companion, 4344. New York: ACM Press.
Hopcroft, J. E., and J. D. Ullman. 1979. Introduction to Automata Theory, Languages and
Computation. Reading, MA: Addison-Wesley.
Hutchins, E. L., J. D. Hollan, and D. A. Norman. 1986. Direct manipulation interfaces. In
User-Centered System Design, ed. D. A. Norman and S. W. Draper, 87124. Hillsdale, NJ:
Laurence Erlbaum.
Jakobson, R. 1960. Linguistics and poetics. In Style in Language, ed. T. A. Sebeok, 350377.
Cambridge, MA: The MIT Press.
Johnson, D. G. 2001. Computer Ethics, 3rd ed. Upper Saddle River, NJ: Prentice Hall.
Jrgensen, H. D. 2001. Interaction as a framework for exible workow modelling. In Pro-
ceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group
Work, 3241. New York: ACM Press.
Josephson, J. R., and S. G. Josephson. 1994. Abductive Inference. Computation, Philosophy,
Technology. New York: Cambridge University Press.
Kammersgaard, J. 1988. Four different perspectives on human-computer interaction. Inter-
national Journal of Man-Machine Studies 28(4): 343362.
Kaptelinin, V. 1996. Activity theory: Implications for human-computer interaction. In
Context and Consciousness, ed. B. A. Nardi, 103116. Cambridge, MA: The MIT Press.
Kaptelinin, V., B. A. Nardi, and C. Macaulay. 1999. The activity checklist: A tool for repre-
senting the space of context. Interactions 6(4): 2739.
Kirsh, D., and P. Maglio. 1995. On distinguishing epistemic from pragmatic action. Cogni-
tive Science 18: 513549.
Lakoff, G. 1987. Women, Fire, and Dangerous Things. Chicago: University of Chicago Press.
Lakoff, G. 1988. Cognitive semantics. In Meaning and Mental Representations, ed. U. Eco,
M. Santambrogio, and P. Violi, 119154. Bloomington: Indiana University Press.
Lakoff, G., and M. Johnson. 1981. Metaphors We Live By. Chicago: University of Chicago
University Press.
Landauer, T. 1991. Lets get real: A position paper on the role of cognitive psychology in the
design of humanly useful and usable systems. In Designing Interaction: Psychology at the
Human-Computer Interface, ed. J. M. Carroll, 6073. New York: Cambridge University
Press.
Leech, G. 1983. The Principles of Pragmatics. London: Longman.
Leite, J. C. 1998. Modelos e formalismos para a engenharia semitica de interfaces de
usurio. Ph.D. thesis, Departamento de Informtica, PUC-Rio, Rio de Janeiro.
Bibliographic References 269
Lewis, C., and D. A. Norman. 1986. Designing for error. In User-Centered System Design,
ed. D. A. Norman and S. W. Draper, 411432. Hillsdale, NJ: Laurence Erlbaum.
Lieberman, H., F. Patern, and V. Wulf. Forthcoming. End User Development: Empowering
People to Flexibly Employ Advanced Information and Communication Technology.
Dordrecht: Kluwer Academic Publishers.
Marcus, A. 1992. Design for Electronic Documents and User Interfaces. New York:
Addison-Wesley.
Marx, G. T. 1994. New telecommunications technologies require new manners. Telecom-
munications Policy 18(7): 538551.
McLuhan, M. [1964] 2002. Understanding Media, 2d ed. Cambridge, MA: The MIT Press.
Medway, P., and B. Clark. 2003. Imagining the building: Architectural design as semiotic
construction. Design Studies 24(3): 255273.
Moore, J. 1995. Participating in Explanatory Dialogues. Cambridge, MA: The MIT Press.
Morris, C. 1938. Foundations of a Theory of Signs. Chicago: University of Chicago Press.
Mullet, K., and D. Sano. 1995. Designing Visual Interfaces. Mountain View, CA: SunSoft
Press.
Nadin, M. 1988. Interface design and evaluation. In Advances in Human-Computer Inter-
action, vol. 2, ed. R. Hartson and D. Hix, 45100. Norwood, NJ: Ablex Publishing Corp.
Nadin, M. 1997. Signs and systems. A semiotic introduction to systems design. Available at
http://www.code.uni-wuppertal.de/uk/all_pdf_les/sign+sys.pdf. Accessed January 2004.
Nake, F., and S. Grabowski. 2001. Human-computer interaction viewed as pseudo-
communication. Knowledge-Based Systems 14(8): 441447.
Nardi, B. A. 1993. A Small Matter of Programming. Cambridge, MA: The MIT Press.
Nardi, B. A. 1996. Context and Consciousness. Cambridge, MA: The MIT Press.
Newton, S. 2004. Designing as disclosure. Design Studies 25(1): 93109.
Nielsen, J. 1994. Heuristic evaluation. In Usability Inspection Methods, ed. J. Nielsen and
R. L. Mack, 2562. New York: John Wiley & Sons.
Nielsen, J. 2001. Ten usability heuristics. Available at http://www.useit.com/papers/
heuristic/heuristic_list.html.
Norman, D. A. 1986. Cognitive engineering. In User-Centered System Design, ed. D. A.
Norman and S. W. Draper, 3161. Hillsdale, NJ: Laurence Erlbaum.
Norman, D. A. 1988. The Design of Everyday Things. New York: Basic Books.
Norman, D. A. 1999. Affordance, convention and design. Interactions 6(3): 3842.
Norman, D. A., and S. W. Draper. 1986. User-Centered System Design. Hillsdale, NJ:
Laurence Erlbaum.
Ogden, C. K., and I. A. Richards. 1946. The Meaning of Meaning, 8th ed. New York:
Harcourt, Brace & World.
Patern, F. 2000. Model-based design and evaluation of interactive applications. London:
Springer-Verlag.
270 Bibliographic References
Peirce, C. S. 19311958. Collected Papers of Charles Sanders Peirce, vols. 18, ed. C.
Hartshorne and P. Weiss. Cambridge, MA: Harvard University Press.
Prates, R. O. 1998. A engenharia semitica de linguagens de interfaces multi-usurio. Ph.D.
thesis, Departamento de Informtica, PUC-Rio, Rio de Janeiro.
Prates, R. O., and C. S. de Souza. 1998. On the rationale of interface semiotics for multi-
user applications. In Proceedings of the Joint Conference on the Science and Technology of
Intelligent Systems ISIC/CIRA/ISAS98. Gaithersburg, MD: IEEE Press.
Prates, R. O., S. D. J. Barbosa, and C. S. de Souza. 2000. A case study for evaluating inter-
face design through communicability. In International Conference on Designing Interactive
Systems DIS2000, 308317. New York: ACM Press.
Prates, R. O., C. S. de Souza, and S. D. J. Barbosa. 2000. A method for evaluating the com-
municability of user interfaces. ACM Interactions 7(1): 3138.
Preece, J., Y. Rogers, and H. Sharp. 2002. Interaction Design. London: John Wiley and Sons.
Preece, J., Y. Rogers, H. Sharp, D. Benyon, S. Holland, and T. Carey. 1994. Human-
Computer Interaction. Reading, MA: Addison-Wesley.
Puerta, A., and J. Eisenstein. 1999. Towards a general computational framework for model-
based interface development systems. In International Conference on Intelligent User Inter-
faces IUI99, 171178. New York: ACM Press.
Puerta, A. R. 1996. The MECANO Project: Comprehensive and integrated support for
model-based interface development. In Computer-Aided Design of User Interfaces, ed. Jean
Vanderdonckt, 1925. Namur, Belgium: Presses Universitaires de Namur.
Rappin, N., M. Guzdial, M. Realff, and P. Ludovice. 1997. Balancing usability and learning
in an interface. In Proceedings of CHI 97, 479486. New York: ACM Press.
Reeves, B., and C. Nass. 1996. The Media Equation. Cambridge: Cambridge University Press.
Reppening, A., and T. Sumner. 1995. Agentsheets: A medium for creating domain-oriented
visual languages. Computer 28(3): 1725.
Rogers, Y., L. Bannon, and G. Button. 1994. Rethinking theoretical frameworks for HCI: A
review. SIGCHI Bulletin 26 (1): 2830.
Schank, R. 1975. Conceptual Information Processing. Amsterdam: North-Holland.
Schn, D. A. 1983. The Reective Practitioner. New York: Basic Books.
Schn, D. A., and J. Bennett. 1996. Reective conversation with materials. In Bringing Design
to Software, ed. T. Winograd, 171184. New York: Addison-Wesley.
Searle, J. R. 1969. Speech Acts. Cambridge: Cambridge University Press.
Searle, J. R. 1979. Expression and Meaning. Cambridge: Cambridge University Press.
Sedig, K. 1998. Interface style, ow, and reective cognition: Issues in designing interactive
multimedia mathematics learning environments for children. Unpublished Ph.D. diss.,
Department of Computer Science, The University of British Columbia, Vancouver, Canada.
Sedig, K., M. Klawe, and M. Westrom. 2001. Role of interface manipulation style and scaf-
folding on cognition and concept learning in learnware. ACM Transactions on Computer-
Human Interaction 8(1): 3459.
Bibliographic References 271
Sellen, A., and A. Nicol. 1990. Building user-centered online help. In The Art of Human-
Computer Interface Design, ed. B. Laurel, 144154. Reading, MA: Addison-Wesley.
Shankland, S. 2000. New email software can help you bite your tongue. CNET News.com.
Available at http://news.com.com/2100-1040-245790.html?legacy=cnet. Accessed Sept. 15,
2000.
Shannon, C. E., and W. Weaver. 1949. The Mathematical Theory of Communication. Urbana:
University of Illinois Press.
Shneiderman, B. 1983. Direct manipulation: a step beyond programming languages. IEEE
Computer 16(8): 5769.
Shneiderman, B. 1998. Designing for the User Interface, 3rd ed. Reading, MA: Addison-
Wesley.
Shneiderman, B., S. Card, D. A. Norman, M. Tremaine, and M. M. Waldrop. 2002. CHI@20:
Fighting our way from marginality to power. In CHI 02 Extended Abstracts on Human
Factors in Computer Systems, 688691. New York: ACM Press.
Silveira, M. S. 2002. Metacomunicao designer-usurio na interao humano-computador.
Ph.D. thesis, Departamento de Informtica, PUC-Rio, Rio de Janeiro.
Silveira, M. S., C. S. de Souza, and S. D. J. Barbosa. 2001a. Augmenting the affordance of
online help content. In Interactions without Frontiers, Proceedings of the Joint AFIHM-BCS
Conference on Human-Computer Interaction IHM-HCI 01, vol. 1, 279296. London:
Springer-Verlag.
Silveira, M. S., C. S. de Souza, and S. D. J. Barbosa. 2001b. Semiotic engineering contribu-
tions for designing online help systems. In Proceedings of the 19th Annual International Con-
ference on Computer Documentation, 3138. New York: ACM Press.
Silveira, M. S., C. S. de Souza, and S. D. J. Barbosa. 2003. Um mtodo da engenharia semi-
tica para a construo de sistemas de ajuda online. In Proceedings of the Latin American
Conference on Human-Computer Interaction, 167177. Rio de Janeiro: PUC-Rio.
Simon, H. 1981. The Sciences of the Articial, 2nd ed. Cambridge, Mass.: The MIT Press.
Suchman, A. 1987. Plans and Situated Actions. New York: Cambridge University Press.
Sundar, S. S., and C. Nass. 2000. Source orientation in human-computer interaction. Com-
munication Research 27(6): 683703.
Sutcliffe, A. 2000. On the effective use and reuse of HCI knowledge. ACM Transactions on
Computer-Human Interaction 7(2): 197221.
Wegner, P. 1995. Interaction as a basis for empirical computer science. ACM Computing
Surveys 27(1): 4548.
Wharton, C., J. Rieman, C. Lewis, and P. Polson. 1994. The cognitive walkthrough method:
A practitioners guide. In Usability Inspection Methods, ed. J. Nielsen and R. L. Mack,
105140. New York: John Wiley and Sons.
Winograd, T. 1987. A language-action perspective on the design of cooperative work.
Human-Computer Interaction 3(1): 330.
Winograd, T., and F. Flores. 1986. Understanding Computers and Cognition. Reading, MA:
Addison-Wesley.
272 Bibliographic References
Software and Online Databases and Services
Active Worlds. Activeworlds Incorporated.
Acrobat 5.0.5. Adobe Systems Incorporated.
Amazon.com, Inc. Available online at http://www.amazon.com.
Arachnophilia 3.9. 19961998, Paul Lutus.
DOS. Microsoft Corporation.
Eudora 3.6 Light, Eudora 5.0.2. Qualcomm Incorporated.
HCI Bibilography: Human-Computer Interaction Resources. Available online at
http://www.hcibib.org/. Accessed December 2003.
Lotus 1-2-3 (9). Lotus Development Corporation. 19881998
Lotus Freelance Graphics (9). Lotus Development Corporation. 19881998
Lotus SmartSuite (9). Lotus Development Corporation. 19881998
Lotus WordPro (9). Lotus Development Corporation. 19881998
Mac OS Finder. Microsoft Apple Computer, Inc.
Microsoft Windows 98 and Windows 2000. Microsoft Corporation.
Microsoft Excel 2002. Microsoft Corporation.
Microsoft Ofce 2002. Microsoft Corporation.
Microsoft Outlook 2002. Microsoft Corporation.
Microsoft PowerPoint 2002. Microsoft Corporation.
Microsoft Word 2002. Microsoft Corporation.
MSN Messenger 5.0. Microsoft Corporation.
Opera 7.11. Opera Software ASA.
Pine 4.58a Program for Internet News & Email. Computing & Communications at the
University of Washington.
PhoneWorks 2002. RingCentral, Incorporated.
SmartFTP v1.0.979. SmartFTP. Available online at http://www.smartftp.com. Accessed
December 2003.
StageCast
TM
Creator 1.0. StageCast Software, Inc.
StuffIt Expander. Alladin Systems, Inc.
Super Tangrams. E-GEMS and University of British Columbia.
SWI Prolog 5.2.6. 19902003 University of Amsterdam. Available online at
http://www.swi-prolog.org/. Accessed December 2003.
The Coordinator. Action Technologies, Inc.
The Peirce Edition Project | EP 1. Introduction. Indiana UniversityPurdue University Indi-
anapolis. Available online at http://www.inpui.edu/~peirce/. Accessed December 2003.
Treemap 3.2. HCIL. University of Maryland College Park. Available online at
http://www.cs.umd.edu/hcil/Treemap/. Accessed December 2003.
Unix 4.1. The Open Group.
Visual Basic for Applications 6.3. Microsoft Corporation.
Windows 98 CD Player. Microsoft Corporation.
Windows Media Player 9. Microsoft Corporation.
Windows NetMeeting 3.0. Microsoft Corporation.
WinVi 2.71. Raphael Molle 19941999. Available online at http://www.winvi.de/en/.
Accessed December 2003.
WinZip 8.1. WinZip Computing, Inc.
274 Software and Online Databases and Services
Clark, H. H., 101
Cunha, C. K. V., 198
Cypher, A., 183
Cypher, A., and D. Caneld-Smith, 172,
183
Danesi, M., and P. Perron, 3, 59
da Silva, E. J., C. S. de Souza, R. O. Prates,
and A. M. Nicolaci-da-Costa, 239
da Silva, S. R. P., 197
de Paula, M. G., 155
de Saussure, F., 36
de Souza, C. S., 4, 24
de Souza, C. S., and J. Preece, 232
de Souza, C. S., and K. Sedig, 52, 55
de Souza, C. S., R. O. Prates, and S. D. J.
Barbosa, 159
de Souza, C. S., R. O. Prates, and T. Carey,
10, 91, 141
de Souza, C. S., S. D. J. Barbosa, and S. R.
P. da Silva, 195
Delio, M., 64
DiGiano, C., and M. Eisenberg, 183
Dourish, P., 193, 217, 231, 232, 241,
248
Dourish, P., J. Holmes, A. MacLean, P.
Marqvardsen, and A. Zbyslaw, 198
Eco, U., 3, 26, 35, 3940, 42, 59, 65,
7173, 75, 97, 196, 245
Eco, U., M. Santambroggio, and P. Violi,
39
Author Index
Ackerman, M., 217
Alexander, C., 75
Andersen, P. B., 4, 24, 197
Andersen, P. B., B. Holmquist, and J.
Jensen, 4, 93
Aristotle, 46, 80
Austin, J. L., 59, 101
Barbosa, C. M. A., 235
Barbosa, S. D. J., 193
Barbosa, S. D. J., and C. S. de Souza, 180,
194
Barbosa, S. D. J., and M. G. de Paula, 155
Barnard, P., J. May, D. Duke, and D. Duce,
31
Bastien, J. M. C., D. Scapin, and C. Leulier,
146
Bellotti, V., S. Buckingham Shum, A.
MacLean, and N. Hammond, 9
Bdker, S., 112, 153
Bowers, J., and J. Churcher, 225
Braa, K., and R. Vidgen 9
Broadbent, G., 248
Brown, G., 59
Brown, J. S., and P. Duguid, 113, 149
Card, S. K., T. P. Moran, and A. Newell,
32, 153, 255
Carroll, J. M., 151, 254
Carroll, J. M., and M. B. Rosson, 151
Cawsey, A. J., 129
Chandler, D., 35, 36, 40
276 Author Index
Ellis, C., S. Gibbs, and G. Rein, 215, 235,
239
Farkas, D. K., 151, 158
Fetzer, J. H., 44
Fisher, G., 112
Fitt, P. M., 9
Flores, F., M. Graves, B. Harteld, and T.
Winograd, 5960
Garcia, A. C. B., and C. S. de Souza, 249
Gelernter, J., and S. Jagganathan, 74
Gibson, J. J., 10
Golightly, D., and D. Gilmore, 55
Grice, H. P., 61, 233
Gruber, T. R., 95
Gugerty, L., 31
Halasz, F. G., and T. P. Moran, 201
Hayakawa, S. I., 101
Hollan, J., E. Hutchins, and D. Kirsh, 31,
91
Holst, S. J., 55
Hopcroft, J. E., and J. D. Ullman, 151
Hutchins, E. L., J. D. Hollan, and D. A.
Norman, 4849, 90, 100, 214
Jakobson, R., 65, 87
Johnson, D. G., 223
Jrgensen, H. D., 198
Josephson, J. R., and S. G. Josephson, 39
Kammersgaard, J., 24
Kaptelinin, V., 214
Kaptelinin, V., B. A. Nardi, and C.
Macaulay, 91
Kirsh, D., and P. Maglio, 33
Lakoff, G., 201, 205
Lakoff, G., and M. Johnson, 7980
Landauer, T., 250
Leech, G., 6061, 233
Leite, J. C., 24
Lewis, C., and D. A. Norman, 63, 104,
230
Lieberman, H., F. Patern, and V. Wulf,
182
Marcus, A., 248
Marx, G. T., 226, 239
McLuhan, M., 97
Medway, P., and B. Clark, 248
Moore, J., 129, 151, 161
Morris, C., 40, 57, 104
Mullet, K., and D. Sano, 74, 113, 146, 248
Nadin, M., 4, 24, 40, 248, 250
Nake, F., and S. Grabowski, 28
Nardi, B. A., 183, 190, 215
Newton, S., 248
Nielsen, J., 61, 143
Norman, D. A., 10, 89, 91, 111, 152, 163,
214
Norman, D. A., and S. W. Draper, 7
Ogden, C. K., and I. A. Richards, 99
Patern, F., 153
Peirce, C. S., 38, 40
Prates, R. O., 235
Prates, R. O., and C. S. de Souza, 235
Prates, R. O., C. S. de Souza, and S. D. J.
Barbosa, 86, 113
Prates, R. O., S. D. J. Barbosa, and C. S. de
Souza, 139
Preece, J., Y. Rogers, and H. Sharp, 143
Preece, J., Y. Rogers, H. Sharp, D. Benyon,
D., S. Holland, and T. Carey, 215
Puerta, A., and J. Eisenstein, 153
Puerta, A. R., 153
Rappin, N., M. Guzdial, M. Realff, and P.
Ludovice, 55
Reeves, B., and C. Nass, 93, 252
Reppening, A., and T. Sumner, 172
Rogers, Y., L. Bannon, and G. Button, 9,
250
Schank, R., 58
Schn, D. A., 31, 105, 185, 248
Author Index 277
Schn, D., and J. Bennett, 32
Searle, J. R., 59, 101
Sedig, K., 54
Sedig, K., M. Klawe, and M. Westrom, 52,
54
Sellen, A., and A. Nicol, 129, 158
Shankland, S., 63
Shannon, C. E., and W. Weaver, 65
Shneiderman, B., 10, 21, 50, 61, 112, 214
Shneiderman, B., S. Card, D. A. Norman,
M. Tremaine, and M. M. Waldrop, 3132,
250, 251
Silveira, M. S., 153, 157
Silveira, M. S., C. S. de Souza, and S. D. J.
Barbosa, 153, 160
Simon, H., 32, 248
Suchman, A., 33, 112, 153
Sundar, S. S., and C. Nass, 93
Sutcliffe, A., 31
Wegner, P., 44
Wharton, C., J. Rieman, C. Lewis, and P.
Polson, 141
Winograd, T., 59, 60
Winograd, T., and F. Flores, 33, 151, 216,
225, 241, 249
Subject Index
semiotic proling in, 146150
utterances for, 129139, 158
Communication, 29, 6575
denition of, 26
Communication process, 72
Communicative breakdowns, 115126, 137
Communicative competence, 212
Communicative functions, 6571, 148
conative, 66
expressive, 66
metalinguistic, 69
phatic, 68
poetic, 71
referential, 68
Communicative innovation, 179
Computer as medium, 88, 93
Computer as source, 94
Computer as storyteller, 256
Computer languages
a taxonomy of communicative
manipulations of, 175
a taxonomy of formal manipulations of,
167
semiotic dimensions of, 166
Computer literacy, 152
Computer-mediated communication, 24,
31, 88, 218
breakdowns in, 227, 229
Conceptual dependency, 58
Content, 71, 98
Context, 88
Conversation with materials, 185190
Abduction, 4244, 192
abductive reasoning and, 149, 194
denition of, 39
the MS Excel example of, 4446
Activity theory, 91, 111, 153, 214
Affordances, 10, 91, 113
Agreement maxim, 62
Analogies, 201
Approbation maxim, 62
Articulatory directness, 4852
Articulatory distance, 99
Articial intelligence, 39, 58, 153
Avatars, 212
Beliefs, 247
Biotic, 57, 89
Branding, 148
Channel, 88
Code, 88
Cognitive engineering, 89, 100, 163
Cognitive theories, 111
Cognitive walkthrough, 141
Communicability, 113114, 210
breakdown categories in, 124, 137
denition of, 113
the Eudora example of, 115120
Communicability evaluation, 115, 185
discrimination principle for, 124
interpreting tagged interaction in, 139
146
the method of, 126150
280 Subject Index
Culture, 29, 40, 59, 62, 100, 113
social conventions in, 213
Customizable applications, 164
Customization, 104, 165, 167, 173, 232
Designers deputy, 24, 8995, 112
denition of, 90
the explanatory discourse of the, 150160,
191, 197, 211
reied and anthropomorphized forms of
the, 90
Designer-user communication, 8589
Design intent, 30, 113
Design patterns, 191
Design rationale, 107, 148, 153
Direct manipulation, 10, 55, 205, 214
the CD Player example of, 7579
discursive competence and, 75, 90
and secondness, 52
the Treemap example of, 4952
Distributed cognition, 91
Emoticons, 221
End-user programming, 104
application-centered vs. user-centered
approaches to, 190
communicative capacity expansions for,
175184
extending software through gurative
speech as a form of, 179, 193194
formal language expansions in, 166175
the PowerPoint macro example of,
187190
the semiotic engineering approach to,
163166
type-controlled metalinguistic
manipulations in, 194197
users communication about extensions
and, 197198
Epistemic tools, 1113, 106108
for building customizable and extensible
applications, 198
for building explanatory discourse, 160
for building multi-user applications, 242
denition of, 33
for users (in customization and extension
tasks), 190
for users (in general), 253
Epistemology of practice, 105
Ergonomic criteria, 146
Ethics, 222, 229
Explanation systems, 129
Expression, 71, 98
Expressive code, 114, 121
Expressive democracy, 222
Extension, 165, 167, 173, 232
Figurative speech, 180
Firstness, 46, 49, 212, 213, 221
Formal language, 151
Generative theories, 251
Generosity maxim, 62
Grammar, 74, 81
Gricean maxims, 61, 148, 152, 233
HCI metrics, 99104
Help!, 138
denition of, 130
Heuristic evaluation, 143, 145146
Human-computer interaction as design,
185
Human information processor, 32
I can do otherwise, 120, 138, 179,
185186, 225
denition of, 136
I cant do it this way, 119, 138, 185
denition of, 133
I give up, 138
denition of, 134
Iconic representations, 221
Icons, 40
denition of (see Firstness)
Illocutionary acts, 59, 123, 147, 154, 175
Illocutionary force, 237238
Impermeable signs, 165, 167
Indexical representations, 219
Indices, 40, 56
denition of (see Secondness)
Subject Index 281
Intellectual artifacts, 921
denition of, 10
Intelligent agents, 58
Intelligent user interfaces, 108
Intent, 59, 98, 176177, 185
Interaction model, 155
Interaction time, 84, 152
Interactive idiolect, 139, 179
Interactive tokens, 113, 186
Interactive types, 113, 186
Interface agents, 187, 204, 215
Interpretant, 40, 56
Interpretive abstraction principle, 195
Interpretive code, 114, 121
Introducing technology
the Acrobat example of, 1321
a shift in HCI goals for, 2223
Language-action perspective, 59, 225,
228
Langue, 82
Looks ne to me, 120, 138
denition of, 133
Macros, 168
and macro programming, 168, 171, 178,
194
and macro recording, 168, 178
Marq-G*, 235
Maxim of manner, 61
Maxim of quality, 61
Maxim of quantity, 61
Maxim of relation, 61
Meaning, 26, 35, 38, 40, 58, 87, 99, 101,
136, 245247
Meaning expansion, 163
Meaning negotiation, 151152, 160
Medium, 252
Medium is the message, the, 97
Message, 252
Metacommunication, 83, 146
the Arachnophilia example of, 8587
stages of, 2425
Metacommunication artifacts, 24
Metacommunication message
in customizable and extensible
applications, 191
the generic form or the, 84
in multi-user applications, 210
Metalinguistic function, 150. See also
Communicative functions
Metaphors, 163
Metaphors and metonymies, 7981, 149,
179181, 193, 201
the MS Word example of, 8182
Minimalism, 151152, 158
Model-based design, 153, 193, 197
Modesty maxim, 62
Multi-user applications
the communication center metaphor in,
203
conceptual metaphors in, 201209
manners in, 219, 226
a semiotic inspection of, 217232
speech acts in, 237238
the telecommunications device metaphor
in, 207
the virtual environment metaphor in, 205
Natural language interfaces, 256
Natural language processing, 58
Object, 40
Online help, 86, 112, 129, 149, 257
building the designers deputy discourse
for, 15360
questions to be addressed by, 157
Online repository
HCI Bibliography, 3
The Peirce Edition Project, 38
Oops!, 117, 138, 224
denition of, 132
Operational level of interaction, 125, 133,
141
Paradox of the active user, 151
Parole, 82
Pattern languages, 75
Patterns, 74
Patterns of change, 104
282 Subject Index
Perlocutionary acts, 59, 123, 147, 154, 175
Phenomenological categories. See Firstness;
Secondness; Thirdness
Politeness, 226
Politeness in HCI
the Macintosh Finder example of, 64
the Moodwatch example of, 63
Politeness principle, 62, 233
Pragmatics, 89
and culture, 5763
denition of, 57
Principles versus rules, 61, 63
Problem framing, 185
Programming by demonstration, 183
Pruning, 173
Receiver, 87
Redundancy, 233
Referent, 40, 47
Reection in action, 3134, 123, 211, 248,
250
Representamen, 40
Requirements analysis, 103
Resignication, 112, 213, 241
Scientic fragmentation, 254
Scripting, 171, 178, 194
Scripting language, 196
Secondness, 47, 49, 183, 212
Semantic directness, 4852
Semantic distance, 99
Semiology, 3638
arbitrariness of signs in, 3738, 48
and HCI, 3738
langue as dened by, 36
parole as dened by, 36
sign structure in, 40
Semiosis, 98, 112, 149
denition of, 4142
Semiotic continuum principle, 195
Semiotic engineering
and its contribution to HCI, 23, 33
the design space for, 87
an epistemology for, 28, 99104
a methodology for, 28, 1045
an ontology for, 27, 9599
as a reective theory, 29
a research agenda for, 25859
theoretical goals of, 45, 83
theoretical limitations of, 2629
the unit of analysis in, 29
and user-centered design, 79
Semiotic gulf, 163
Semiotic innities, 164, 231
Semiotics, 35
and computation, 245
and culture, 3
denition of, 3
and design, 248
and HCI, 45
as an instrument of science, 40, 104
as the logic of culture, 40, 59
the Peircean perspective on, 3840
Sender, 87
Sign classications, 4648
and cognitive engineering metrics, 48
the Super Tangrams example of, 5257
Sign function, 71
Sign production, 7275, 120, 148
material continuum in, 73
mode of articulation in, 74
physical labor in, 73
token-type ratio in, 73, 120
Signication, 29
and communication, 206
denition of, 26
Signication codes, 126, 164, 176, 181,
194, 223
Signication system, 105, 114, 141, 148,
157
Ecos denition of, 72
in education, 56
Signifying order, 59, 100, 105
Signs, 27
in HCI, 5
the Peircean denition of, 38
the Peircean structure of, 40
Sociability, 224, 226
the Netmeeting example of, 215216
Social computing, 217
Subject Index 283
Social protocol, 235, 239242
Social-technical gap, 217
Speech acts, 59, 112
assertives, 60, 62
commissives, 60, 62, 234
declaratives, 60
directives, 60, 62, 237238
expressives, 60, 62
Strategic aspects of HCI, 1223
Strategic level of interaction, 125, 258
Structuralism, 37
Symbol processing, 183
Symbolic representations, 220
Symbols, 40, 56
denition of (see Thirdness)
Sympathy maxim, 62
System accountability, 232
Systems identity, 166, 168, 172, 174
Tact maxim, 62
Tactical level of interaction, 125, 133, 141,
258
Task model, 153
Technical rationality, 3134, 248
Technological protocol, 235, 239242
Thanks, but no, thanks, 138, 179, 186,
225
denition of, 135
Thirdness, 47, 49, 55, 183, 212, 220
Trigger theories, 251
Trust, 93
Turing machine, 44, 74, 98
Unlimited semiosis, 2628, 41, 77, 176,
185, 256
User-centered design, 96, 99
Verbalizable applications, 197
What happened?, 138
denition of, 131
What now?, 138
denition of, 131
Whats this?, 138
denition of, 129
Where am I?, 138, 180
denition of, 132
Where is it?, 117, 119, 138
denition of, 131
Why doesnt it?, 138, 188
denition of, 130
Workow systems, 198, 231
Software Index
StuffIt Expander, 6970
Super Tangrams, 5257
SWI Prolog, 6667, 187
Treemap, 1213, 23, 4952
Unix, 139, 180
Visual Basic for Applications, 189190
Windows, 48, 68, 115, 129, 130
Windows 98 CD Player, 7579, 102
Windows Media Player, 7071
Windows NetMeeting, 69, 214216,
224225, 236239
WinVi, 1112, 23
Winzip, 186
Acrobat, 1323, 258
Active Worlds, 205207, 212
Amazon.com, 9394
Arachnophilia, 8586
The Coordinator, 5960, 225, 234
DOS, 115, 178179, 181
Eudora, 6367, 115122, 127128, 134,
149, 154
Lotus 1-2-3, 190
Lotus Freelance Graphics, 190
Lotus SmartSuite, 190
Lotus WordPro, 190
Mac OS Finder, 64
Microsoft Excel, 4446, 190
Microsoft Ofce, 190, 204205, 215
Microsoft Outlook, 204205
Microsoft PowerPoint, 187190
Microsoft Word, 8182, 190
MSN Messenger, 203204, 207, 219220
Opera, 92
PhoneWorks, 207209, 213
Pine, 139, 180, 193
SmartFTP, 56
StageCast Creator, 184