Visualization Viewpoints

Editor:
Theresa-Marie Rhyne

Toward a Perceptual Theory

of Flow Visualization
Colin Ware
University
of New
Hampshire
W hat constitutes good visualization re-
search? Are researchers aiming at better
algorithms, theory, or design guidelines
to produce better visualizations?
One way to improve visualization research is to
many visual pattern queries in flow visualization,
such as finding locations of critical points, areas of
high turbulence, and advection pathways.

Mapping hypothesis
model it on natural and physical sciences and de- To illustrate how perceptual theory can be fruit-
velop a body of theory. A good theory should make fully applied to a common visualization problem,
testable predictions about which data mapping dis- I focus mainly on a single task: judging advection
plays will be most effective, letting researchers gain pathways in steady flow. To perceive an advec-
insight about their data (exploration) or help others tion pathway we must perceptually trace a path
understand what the data shows (explanation). Hu- from a starting point in the flow. This suggests
man perception theory, pragmatically applied, should that the best representation of an advection path-
lead to a substantial testable visualization theory. way will be a visual contour because the brain has
Perceptual theory has much to tell us about ef- mechanisms to rapidly find contours marking the
fective flow visualization. Since visualization is a boundaries of objects or linear features in the en-
practical tool, a meaningful theory must be con- vironment.2 Whatever lets us perceive pathways as
structive—not just descriptive—leading to better contours anywhere in a flow field is likely to be the
data representations. I contend that such a use- most effective graphic design to support judgments
ful body of theory exists and though it might be about advection pathways. I call this proposition
incomplete it still can be useful. Major gaps in my mapping hypothesis. It is worth noting that
the theory provide promising avenues for future there are alternatives. For example, the orientation
research in a multidisciplinary program that can of flow has been represented by a cycle of colors or
benefit both the human perception field and the by the actual advection of scattered random points
applied science of data visualization. seeded through the flow field. Nevertheless, using
The proposition that visualization lets us perceive contour orientation to reveal flow direction is the
patterns in data (and hence discover meaning) most common technique.
seems uncontroversial; this leads to a body of sci- Almost all static flow visualization methods
ence that bears directly on the problem but is rarely generate contours that are tangential to the di-
taken into account. Modern neuropsychology has rections of flow (see Figure 1). The most common
much to say about visual pattern perception. This method is to use a grid of little arrows. Some
discipline has made great strides over the past sev- methods use curved arrows; others use continu-
eral decades, driven by advances in psychophysics, ous contours.3,4 Line integral convolution typically
single-cell visual recordings of the brains of ani- produces blurred contours.5
mals, and functional magnetic resonance imaging,
which reveals the parts of the brain using the most Contour perception
oxygen and, hence, presumed to be the most active A well-developed body of perceptual theory re-
at a given time. lates to contour perception. Since the pioneer-
The patterns we are interested in when we look ing work of Hubel and Wiesel6 in the 1960s, we
at flow vary depending on what aspects of the data know that the primary visual cortex (called V1)
we wish to analyze. For visualization to be a useful at the back of the brain contains large numbers
tool, aspects of this cognitive task must be trans- of neurons, each responding to a small patch of
formed into visual-pattern queries.1 We can make visual space and each selectively tuned in terms

 March/April 2008 Published by the IEEE Computer Society 0272-1716/08/$25.00 © 2008 IEEE
(a) (b) (c)

Figure 1. A grid of arrows is the most common way to visualize a flow field. Each arrow’s tail is tangential to the flow.

of a pattern’s orientation to which the neurons

most strongly respond. One mathematical model
of filtering operation is the Gabor function (see
Figure 2), the product of a sine wave grating and a
Gaussian function. These neurons respond strong-
ly when either an edge or a line is oriented with
the receptive field of the neuron and weakly, or
not at all, when the edge or line is not so aligned.
These orientation filtering neurons are arranged in
a specific architecture called “hypercolumns.” As
1
you progress into the cortex down a hypercolumn 3
d
the receptive fields grow. As you progress laterally an 4A
2 4B
across the cortex, you find the different orien-
4C
tations for that same part of visual space. As a 5
Cortical layers 6
whole, the primary V1 operates as a set of parallel
filters, with hundreds of millions of neurons op-
erating in parallel so that every part of the visual Figure 2. The primary visual cortex contains neurons that respond to
field is simultaneously processed for every orienta- oriented patterns in particular parts of the visual field. Each part of
tion and size of elementary feature. visual space is processed for orientation and size information. The gray
These simple orientation-detecting neurons have image (upper right corner) shows a set of receptive fields of individual
been found in every “higher” animal and are the neurons, each responding best to a particular feature, size, and
basis for all theories of contour detection. The sig- orientation.
nals from individual neurons, however, are not
sufficient to account for the perception of long con-
tinuous contours that might make up an advection
-
pathway. They can signal only the local orientation - -
- -
at each part of a contour. -
The neural mechanisms that bind multiple indi- + +
vidual neurons firing in response to different sec- - -
tions of an extended contour are more speculative, - -
- -
but most theorists agree on something like the fol-
lowing: Individual neurons within V1 are recipro-
cally connected with other neurons also within V1. (a) (b)
Neurons in spatial proximity and aligned with one
another, as shown in Figure 3, have mutually excit- Figure 3. Neurons that have spatially aligned receptive fields mutually
atory connections. Neurons that are in proximity reinforce one another. (a) They also inhibit other nearby neurons. (b)
and not so aligned inhibit one another. The result When a smooth continuous contour (or edge) is imaged in the eye, the
of such a network is that neurons that are stimu- neurons along its path excite, while nearby neurons not on the edge
lated by the image of a continuous line or edge will become inhibited.
fire more strongly than neurons that are stimulated
by little fragments of edges such as might occur in This basic theory of contour perception is already
a rough texture. Some theorists have also proposed sufficient to make a number of straightforward pre-
that neurons responding to an edge rapidly come to dictions about which visualization methods will re-
fire in synchrony with one another.7 sult in better advection perception.

IEEE Computer Graphics and Applications 

Visualization Viewpoints

+
+ +
(a) (b) (c) (d)

Figure 4. A set of methods, a through d in increasing effectiveness, which have been used for flow visualization
(top row). Note that there are no arrowheads here and so the direction of flow is ambiguous. The predicted
effect of the patterns generated by each method on orientation selective neurons (bottom row).

Orientation
of a Macaque monkey to patterns similar to Figure
Direction
4a.9 Although the neurons responded to these pat-
Vector sign
terns, they responded more strongly to ratings of
Representation continuous contours similar to the pattern shown
of a vector
in Figure 4d.
Magnitude
(speed)
Vector sign
The theory presented thus far is incomplete in a
Figure 5. A vector can be broken into three critical respect. The advection pathway for the pat-
components. Many representation methods only terns shown in Figure 4 are ambiguous. It’s nor-
show a subset of these. mal to decompose a vector into a direction and a
magnitude (speed). For purposes of understanding
flow visualization, it is convenient to further de-
Figure 4 illustrates, in simplified form, a set of compose the direction component into an orienta-
methods that have been used to illustrate flow pat- tion and a vector sign, as shown in Figure 5. Many
terns. The theory predicts that the rank order of flow visualization methods such as line integral
effectiveness of these methods will be a, b, c, d for convolution (Figure 1c) clearly show the orienta-
the following reasons: Method b should be bet- tion, but fail to show the vector sign.
ter than method a because a short line segment What does the theory of perception tell us
will give a stronger signal to orientation-sensitive about how to represent the vector sign? To rep-
V1 neurons than will the pairs of dots shown in resent this single bit of information, there must
method a. Method c should be better than method be directional asymmetry along the direction of
b because having the line segments aligned will the contour. The most common device is to use
produce mutual excitation as described. Method d an arrowhead. However, a number of other pos-
should be better than method c because a continu- sibilities exist, some of which Figure 6 illustrates.
ous contour will produce stronger mutual excita- One possible neural mechanism for detecting
tion than broken but aligned contours. this kind of asymmetry is through complex and
Although they were not testing this theory, Laid- hypercomplex cells found in visual areas 1 and
law and colleagues8 performed a study that relates 2.6 These are sometimes called end-stopped cells
to cases in Figures 4b and 4c. They compared jit- because they respond most strongly to oriented
tered arrows with head-to-tail aligned arrows (as features that terminate in the receptive field and
well as other methods) and found that the head- respond weakly, or not at all, to features that
to-tail aligned method produced reduced error for are extended through it (Figure 7). Heider and
advection perception. However, the two cases also colleagues reported that 50 percent of such end-
used different styles of arrows, which may have stopped cells responded asymmetrically, respond-
been a factor. Also relevant is a study that mea- ing more strongly if the feature terminates in a
sured the responses of neurons in the visual cortex particular direction.10 No studies have tested

 March/April 2008
the responses of end-stopped cells to patterns *
like those in Figure 6. However, it seems plau-
sible that some will yield stronger asymmetric
responses than the conventional arrowhead. In
particular, the gray ramp pattern marked with an Figure 6. Ways to create along-contour asymmetry for showing the
asterisk in Figure 6 has very strong asymmetry. vector sign.
One end completely lacks a distinct termination,
so this might be a good choice for indicating the
vector sign, at least with static patterns. Fowler (a)
and Ware introduced patterns like this for flow
visualization and showed that they were unam-
biguously read with respect to the vector sign of
simple flow patterns.11 (b)
Unfortunately, we don’t have a model of a typi-
cal asymmetrical end-stopped neuron. If we did, it
would be possible to make predictions about which
pattern is better. Thus, we can identify an open (c)
question in the perceptual theory of flow visual-
ization, namely, how can we best represent vector
sign information while at the same time preserv-
(d)
ing the perception of flow pathways? Although the
++
gray ramp pattern might be a strong representation
of the vector sign, it fails to represent the advec-
tion pathway with a clear contour. These patterns,
(e)
when arranged head to tail, also might not be as +
effective in stimulating the mutual reinforcement
between simple (symmetric) cell responses that
seem desirable for contour perception (see Figures (f)
7e and 7f). This provides an interesting research
challenge: how to optimally represent the stream- Figure 7. Asymmetric end-stopped neurons that respond to a left-
lines and the vector sign in a dense pattern that hand terminus are shown in green. Those responding to a right-hand
can show as much detail as possible. terminus are shown in red. (a) The level of responding is illustrated
by the little bars. (b) End-stopped neurons don’t respond to contours
Multiple-flow layers passing through their receptive field. (c) They do respond to contours
Another challenge is to represent two layers of that terminate in the receptive field and some do so asymmetrically,
flow simultaneously. Ocean currents, for example, not responding to a termination from the other direction. (d) An arrow
are often stratified, flowing in one direction near symbol provides stronger stimulation at the arrow head than the tail. An
the surface and another direction at greater depth. along-contour gray ramp provides greater asymmetry. When arranged
Perceptual theory again points to some possible head-to-tail, (e) arrows provide better contour continuity (f) than gray
solutions. Recall that simple cells in the cortex ramps because of reinforcement between simple neurons.
have a columnar organization. Neurons deeper
in the cortex respond to larger oriented features
whereas those near the surface respond to fine de-
tail. Psychophysical studies have shown that the
responses of different cortical layers are somewhat
independent. Indeed, the visual system is some-
times said to have spatial frequency “channels”
separating out different sizes of features.12 We
might therefore use different channels to apply to
different layers of a flow. An example from Urness
and colleagues13 illustrates this nicely (although
the authors were not inspired by the theory) and Figure 8. Illustrations from Urness and collegues showing methods for
this is reproduced as Figure 8. displaying overlapping layers.13

Elements of perceptual theory pattern-finding in data is a goal. Allowing data-

You can apply the approach I outline here to revealing patterns to be clearly seen is a funda-
any visualization design problem where support of mental purpose of visualization, which suggests

IEEE Computer Graphics and Applications 

Visualization Viewpoints

that the application of a perception-based ap- Comparing theories

proach can be quite broad. Where does perception-based theory stand with
The proposed approach has the following major respect to other approaches to visualization the-
elements: ory? Clearly it can’t stand alone. Algorithms are
required to generate the necessary mappings be-
■ Define an analytic task to be carried out using visu- tween data and its visual representation and to do
alization. This task must be cognitively executable so efficiently. This brings in the fields of computer
by means of a visual patterns search. The example graphics, numerical algorithms, and databases.
I’ve given here is advection pathway perception. The discipline of design is also essential. Most
■ Propose a mapping (or a set of mappings) between visualizations involve the representation of many
the data and its visual representation. This can variables, and not all can be made to be maximally
consist of an arbitrarily complex algorithm. distinct. The choice of a set of colors for one class
However, it is generally desirable that the map- of data objects constrains the colors that can be
ping be transparent in the sense that it’s easy used to represent others. A good design is usually
for the analyst to understand the relationship a complex optimization problem; the choice of
between the data and its representation. which variables should be mapped to color, which
■ Construct a perceptual theory regarding the ease to texture, and which to motion is generally a
with which the task-relevant patterns can be found matter of judgment rather than science.
as a result of visual search. This theory will nor-
mally be an adaptation of existing perceptual Conclusion
theory applied to particular patterns produced Much of what I’ve said here is implicit in the work
by the mappings under consideration. Use this of many researchers who have applied perceptual
theory to construct testable hypotheses regard- theory to visualization. The purpose of this article
ing either the perceptual efficiency of one map- has been to make the argument explicit.
ping versus another or the optimal values of Currently, most researchers in visualization pay
parameter settings associated with a mapping. very little attention to vision science. The excep-
■ Test the theory. Design display algorithms that tion is when the effective use of color is the sub-
optimally create the mappings under consider- ject. Little research in flow visualization includes
ation according to the theory. Conduct experi- a discussion of the related perceptual theory. Nor
ments to compare how easily you can carry out does it include an evaluation of effectiveness of
visual reasoning tasks about the data under the display techniques that are generated. This is
various mappings and parametric variation. For so, despite Laidlaw’s paper showing that such an
example, Laidlaw created a test to evaluate ad- evaluation is relatively straightforward.
vection pathway perception that involved hav- Of course, it’s not always necessary to relate
ing subjects mark the point where an advected visualization research to perceptual theory. If the
particle, starting in the center of a circle, would purpose of the research is to increase the effi-
cross the boundary of the circle.8 ciency of an algorithm, then the proper test is one
■ Use perceptual theory as inspiration to generate of efficiency, not of perceptual validity.
new mappings. A good theoretical understanding But when a new representation of data is the
of which patterns are likely to be easy to per- subject of research, addressing how perceptually
ceive can often lead to insights into new map- effective it is—either by means of a straight-
pings worth investigating. forward empirical comparison with existing
■ Develop new theory where needed. In many areas, methods or analytically, relating the new map-
perceptual theory is inadequate to provide a ping to perceptual theory—should be a matter
clear hypothesis regarding which mappings are of course. A strong interdisciplinary approach,
likely to be the most effective. The patterns that including the disciplines of perception, design,
are of interest in representing data might not and computer science will produce better sci-
occur naturally in the real world. Also perceptu- ence and better design in that empirically and
al theory has often not been fully developed in theoretically validated visual display techniques
a way that relates clearly to the particular pat- will result.
terns used in data visualization. In this case, the
relationship between visualization science and
perceptual science becomes reciprocally benefi- Acknowledgments
cial. Problems of visually representing data may I thank Donald House, David Laidlaw, and Robert
stimulate new basic research in perceptual sci- Kosara for discussions that helped to develop the ideas
ence, and the result can benefit the applied sci- I presented in this article. Funding was provided by
ence of visualization. NSF ITR grant 0324899.

10 March/April 2008
References J. Physiology, Blackwell Publishing, vol. 203, 1969,
1. C. Ware, Visual Thinking for Design, Morgan Kaufman, pp. 237-260.
2008. 13. T. Urness et al., “Strategies for the Visualization of
2. Z. Li, “A Neural Model of Contour Integration in Multiple 2D Vector Fields,” IEEE Computer Graphics
the Primary Visual Cortex,” Neural Computation, and Applications, vol. 26, no. 4, 2006, pp. 74-82.
vol. 10, no. 4, 1998, pp. 903-940.
3. G. Turk and D. Banks, “Image Guided Streamline Contact author Colin Ware at [email protected].
Placement,” Proc. 23rd Conf. Computer Graphics and
Interactive Techniques (ACM Siggraph), ACM Press, Contact department editor Theresa-Marie Rhyne at
1996, pp. 453-460. [email protected].
4. B. Jobard and W. Lefer, “Creating Evenly Spaced
Streamlines of Arbitrary Density,” Proc.
8th Eurographics Workshop Visualization
in Scientific Computing, Eurographics
Press, 1997, pp. 43-56.
5. B. Cabral and C. Leedon, “Imaging
Vector Fields Using Line Integral
Discover a New World in Programming
Convolution,” Proc. 20th Ann. Conf.
Computer Graphics and Interactive
Looking for a career with an innovative company where you can use your
Techniques (ACM Siggraph), ACM Press,
programming skills to make a difference in the world? Become a key
1993, pp. 262-267.
technical member of ESRI’s development team and you’ll be designing
6. D.H. Hubel and T.N. Wiesel, “Receptive
and developing the next generation of our world-leading geographic
Fields and Functional Architecture of
information system (GIS) mapping software.
Monkey Striate Cortex,” J. Physiology,
The Physiological Society, vol. 195, no.
1, 1968, pp. 215-243.
7. D.J. Field, A. Hayes, and R.F. Hess,
“Contour Integration by the Human
Visual System: Evidence for a Local
‘Association Field,’” Vision Research,
Pergamon Press, vol. 33, no. 2, 1993,
pp. 173-193.
8. D.H. Laidlaw et al., “Quantitative
Evaluation of 2D Vector Field
Visualization Methods,” Proc. IEEE
Visualization, IEEE CS Press, 2001, pp.
143-150.
We are seeking software developers with solid core programming skills and
9. W. Bair et al., “The Timing of Response
a passion for inventing new technology. We have opportunities to work
Onset and Offset in Macaque Visual
on everything from database and Web development to graphics, 2D/3D
Neurons,” J. Neuroscience, Society of
rendering, core server technology, cartography, and using Python to create
Neuroscience, vol. 22, no. 8, 2002, pp.
applications, just to mention a few.
8334-8345.
10. B. Heider, V. Meskanaite, and E. Our employees enjoy competitive salaries, exceptional benefits including
Peterhaus, “Anatomy and Physiology of 401(k) and profit sharing programs, tuition assistance, a café complete with
a Neural Mechanism Defining Depth Starbucks coffee bar, an on-site fitness center, and much more. We employ
Order and Contrast Polarity at Illusory 4,000 people worldwide, 1,700 of whom are based at our Redlands head-
Contours,” European J. Neuroscience, quarters, a community ideally located in Southern California.
Blackwell Publishing, vol. 12, no. 11,
Join ESRI and be a part of changing the world.
2000, pp. 4117-4130.
11. D. Fowler and C. Ware, “Strokes for
Representing Univariate Vector Field
Maps,” Proc. Graphics Interface (GI 89), Learn more about ESRI and apply
1989, pp. 249-253. online at www.esri.com/programmers.
12. C. Blakemore and F.W. Campbell, “On
the Existence of Neurons in the Human Copyright © 2007 ESRI. All rights reserved. ESRI, the ESRI globe logo, and www.esri.com, are trademarks, registered trademarks, or service marks
of ESRI in the United States, the European Community, or certain other jurisdictions. Other companies and products mentioned herein may be
trademarks or registered trademarks of their respective trademark owners. ESRI is an Equal Opportunity Employer.
Visual System Selectively Sensitive to the
Orientation and Size of Retinal Images,”

IEEE Computer Graphics and Applications 11