DONALD D.

HOFFMAN

VISUAL INTELLIGENCE
How The Mind Creates Visual Worlds
Copyright °
c Donald D. Hoffman, 1997
All rights reserved
 for
Gery and Melissa
 Preface

A fter his stroke, Mr. P still had outstanding memory

and intelligence. He could still read and talk, and mixed
well with the other patients on his ward. His vision was
in most respects normal—with one notable exception: He
couldn’t recognize the faces of people or animals. As he put
it himself, “I can see the eyes, nose, and mouth quite clearly,
but they just don’t add up. They all seem chalked in, like on
a blackboard . . . I have to tell by the clothes or by the voice
whether it is a man or a woman . . . The hair may help a lot,
or if there is a mustache . . ..” Even his own face, seen in a
mirror, looked to him strange and unfamiliar. Mr. P had lost
a critical aspect of his visual intelligence.
We have long known about IQ and rational intelligence.
And, due in part to recent advances in neuroscience and
psychology, we have begun to appreciate the importance of
emotional intelligence. But we are largely ignorant that there
is even such a thing as visual intelligence—that is, until it is
severely impaired, as in the case of Mr. P, by a stroke or other
insult to visual cortex. The culprit in our ignorance is vi-
sual intelligence itself. Vision is normally so swift and sure,
so dependable and informative, and apparently so effortless
that we naturally assume that it is, indeed, effortless. But the
swift ease of vision, like the graceful ease of an Olympic ice
skater, is deceptive. Behind the graceful ease of the skater
are years of rigorous training, and behind the swift ease of

6
PREFACE 7

vision is an intelligence so great that it occupies nearly half

of the brain’s cortex. Our visual intelligence richly interacts
with, and in many cases precedes and drives, our rational and
emotional intelligence. To understand visual intelligence is
to understand, in large part, who we are.
It is also to understand much about our highly visual cul-
ture in which, as the saying goes, image is everything. Con-
sider, for instance, our entertainment. Visual effects lure us
into theaters, and propel films like Star Wars and Jurassic
Park to record sales. Music videos usher us before surreal
visual worlds, and spawn TV stations like MTV and VH-1.
Video games swallow kids (and adults) for hours on end,
and swell the bottom lines of companies like Sega and Nin-
tendo. Virtual reality, popularized in movies like Disclosure
and Lawnmower Man, can immerse us in visual worlds of un-
precedented realism, and promises to transform not only en-
tertainment but also architecture, education, manufacturing,
and medicine. As a culture we vote with our time and wallets
and, at least in the case of entertainment, our vote is clear.
Just as we enjoy rich literature that stimulates our rational
intelligence, or a moving story that engages our emotional
intelligence, so also we seek out and enjoy visual media that
challenge our visual intelligence.
Or consider marketing and advertisement, which daily ma-
nipulate our buying habits with sophisticated images. Cor-
porations spend millions each year on billboards, packaging,
magazine ads, and television commercials. Their images can
so powerfully influence our behavior that they sometimes
generate controversy—witness the uproar over Joe Camel.
If you’re out to sell something, understanding visual intelli-
gence is, without question, critical to the design of effective
visual marketing. And if you’re out to buy something, un-
derstanding visual intelligence can help clue you in to what’s
being done to you as a consumer, and how it’s being done.
This book is a highly illustrated and accessible introduction
to visual intelligence, informed by the latest breakthroughs
in vision research. Perhaps the most surprising insight that
has emerged from vision research is this: Vision is not merely
a matter of passive perception, it is an intelligent process of
8 PREFACE

active construction. What you see is, invariably, what your

visual intelligence constructs. Just as scientists intelligently
construct useful theories based on experimental evidence,
so your visual system intelligently constructs useful visual
worlds based on images at the eyes. The main difference is
that the constructions of scientists are done consciously, but
those of your visual intelligence are done, for the most part,
unconsciously.
The constructive power of visual intelligence has long fas-
cinated vision researchers. How can vision conjure up the
endless panoply of colors and shapes and motions that we
see about us in the “real” world? How can it morph a mass
of metal into a murdering maniac in the world of Terminator
2? How can it stretch before us a world in three dimensions
when we watch, with special glasses, a 3D movie like Dial M
for Murder? How can it “boldly go where no man has gone
before”, not just through the “strange new worlds” of Star
Trek but through the stranger new worlds revealed by the
cameras of the Hubble telescope and of probes like Voyager
and Pioneer, worlds for which the eye has not obviously been
adapted?
Vision, as we shall see, has divulged many of its secrets
to physicists, neurobiologists, perceptual psychologists, and
researchers in computer vision. But, as we shall also discover,
many interesting secrets are yet to be unveiled. These secrets
of your visual intelligence, both revealed and unrevealed, are
sure to pique in you the same admiration and curiosity that
animate vision researchers.
For some, science is like a “reaper binder” that methodi-
cally mows down wheat: Science methodically mows down
unanswered questions and leaves ever less room for won-
der. But our exploration of visual intelligence will suggest
that science is more like an island: “The larger the island of
knowledge, the longer the shoreline of wonder.” For each
question we answer ten new ones arise, and for each new
probe at nature entire vistas arise.
Our island of knowledge about visual intelligence has grown
immensely in the last two decades. A book-length tour of the
new terrain can hit but a few of the interesting sights. Visual
PREFACE 9

Intelligence hits those sights that may inspire you to linger on

the shoreline of wonder.
Visual Intelligence has several tourists in mind. For those
in marketing, advertising, and graphic design, Visual Intelli-
gence will better acquaint you with your first client: visual
intelligence. Your images must pass the scrutiny of a cus-
tomer’s visual intelligence before they can go on to convince
her rational intelligence of a need and her emotional intelli-
gence of a desire for your product. Visual intelligence is your
path to the head and heart of a customer, and understanding
visual intelligence is key to successfully navigating that path.
For the reader of popular science, Visual Intelligence ex-
plains why your brain devotes billions of its valuable neu-
rons and trillions of its valuable synapses to vision, why each
of your eyes contains within it more computing power than
the fastest supercomputers made today, why you can buy a
chess machine that beats a Master but can’t yet buy a vision
machine that beats a toddler’s vision, why computer vision
is not only possible but is destined soon to be a multi-billion
dollar industry that alters our day-to-day lives no less than
the computer or the telephone. Visual Intelligence will leave
you astonished at what happens when you simply open your
eyes.
For the undergraduate with an undeclared major, beware,
Visual Intelligence might lure you away from a high-paying
career as a doctor or lawyer or CEO, and lure you into a
career as a vision researcher. The unsolved puzzles of visual
intelligence are a worthy challenge for the sharpest of minds
(and the pay isn’t half bad). There is much to be done, and
the field could use your help.
For those building virtual worlds, Visual Intelligence docu-
ments that human vision (and touch and hearing) is the real
creator of virtual reality (VR), and that the role of VR sys-
tems is to “trick” human vision into creating those realities
that you, the VR designer, want created. To build compelling
virtual worlds, one must understand visual intelligence and
how it constructs visual realities.
For lawyers concerned with eye-witness testimony, Visual
Intelligence conveys the latest scientific insights and perspec-
10 PREFACE

tives on the visual processes that underlie such testimony.

The visual reality of an eye witness is a constructed reality.
Understanding the rules by which eye-witnesses construct
visual realities can be critical to the proper evaluation of eye-
witness testimony.
For philosophers interested in the epistemological and on-
tological issues raised by perception, Visual Intelligence pro-
vides an accessible entrance into the latest empirical and
theoretical literature on vision, and suggests that an ideal-
ist reading of this literature can be at least as compelling as
the best physicalist readings.
For vision researchers, faced with a burgeoning literature
from a variety of disciplines, Visual Intelligence provides a
provocative synthesis. Unifying themes can be discerned
in the diversity of our technical results, and contemplating
these themes is not only worthwhile in its own right but can
alter how we conceive and conduct our technical work.
For anyone who is just plain curious about how they see,
or who isn’t curious because they can’t imagine there’s that
much to it, Visual Intelligence is for you.
That should include just about everyone.
As in many scientific fields, mathematics has cropped up
everywhere in vision research—a help to researchers and a
hindrance to laymen. Fortunately, the key discoveries about
visual intelligence can be conveyed without mathematics. To
that end, mathematics is banished to endnotes for those so
inclined. This approach is, of course, not new. The Greek
physician Galen (AD 130–200), for instance, adopted it when
he wrote about vision:
I have explained nearly everything pertaining to the eyes with
the exception of one point which I had intended to omit . . .
since it necessarily involves the theory of geometry . . . [But] I
felt impelled to take up again what I had omitted and add it to
the end of this book.

Also banished to the back are all notes and citations. These
are listed by page number. If you find an intriguing fact or
picture or quote on page 73 and want to know more, chances
are you’ll find more in Note 73 at the back of the book.
PREFACE 11

Many people have helped in the development of Visual In-

telligence: Laura Andes, Joe Arpaia, Margaret Atherton, Kim-
berley Babb, Bruce Bennett, Francis Crick, Alan Gilchrist,
Heiko Hecht, Dieter Heyer, David Hoffman, Loretta Hoff-
man, Nicole Huber, Stan Klein, Larry Maloney, Rainer Maus-
feld, Brian McLaughlin, Alan Nelson, Chetan Prakash, Whit-
man Richards, Scott Richman, Craig Sauer, Armin Schwe-
gler, Stan Schein, Robert Schwartz, Greg Seyranian, Manish
Singh, Carol Skrenes, Dejan Todorović, VaNessa Vollmer, and
many students in my class on the mind-body problem at UC
Irvine, and in Stan Klein’s class on sensation and perception
at UCLA, who read an earlier version of this book for the
class and gave helpful feedback. My editor at Norton, An-
gela von der Lippe, expertly helped me to streamline the text
and broaden its appeal. Thank you all for your help.
Work on this book was supported in part by a sabbatical
leave from the University of California, Irvine, and by a grant
from the Zentrum für interdisziplinäre Forschung of the Uni-
versity of Bielefeld, Germany. Thank you for your support.
 Contents

1. A Creative Genius for Vision 17

2. Inflating an Artist’s Sketch 35

3. The Invisible Surface that Glows 69

4. Spontaneous Morphing 106

5. The Day Color Drained Away 136

6. When the World Stopped Moving 174

7. The Feel of a Phantom 211

8. Peeking Behind the Icons 224

Epilogue 242

Notes 245

References 260

Index 307

13
VISUAL INTELLIGENCE