Information Visualization (2004) 3, 123133

& 2004 Palgrave Macmillan Ltd. All rights reserved 1473-8716 $30.00
www.palgrave-journals.com/ivs

Analyzing perceptual organization in

information graphics

Martin Wattenberg1 Abstract

Danyel Fisher2 We propose a new method for assessing the perceptual organization of
information graphics, based on the premise that the visual structure of an
1
IBM Watson Research Cambridge, Cambridge, image should match the structure of the data it is intended to convey. The core
MA, U.S.A.; 2University of California, Irvine, CA, of our method is a new formal model of one type of perceptual structure, based
U.S.A. on classical machine vision techniques for analyzing an image at multiple
resolutions. The model takes as input an arbitrary grayscale image and returns a
Correspondence: lattice structure describing the visual organization of the image. We show how
Martin Wattenberg, IBM Watson Research, this model captures several aspects of traditional design aesthetics, and we
Cambridge, 1 Rogers Street, Cambridge, MA
describe a software tool that implements the model to help designers analyze
02142, U.S.A.
E-mail: [email protected]
and refine visual displays. Our emphasis here is on demonstrating the models
potential as a design aid rather than as a description of human perception, but
given its initial promise we propose a variety of ways in which the model could
be extended and validated.
Information Visualization (2004) 3, 123133. doi:10.1057/palgrave.ivs.9500070

Keywords: Screen design; software psychology; perceptual organization

Introduction
The design of information graphics remains a poorly understood, hit-or-
miss process. Part of the difficulty is that models for how humans extract
information from visual displays remain incomplete. Indeed, seemingly
minor design variations can have dramatic effects on comprehensibility. As
a result, creating effective displays often requires expensive user tests, time-
consuming redesigns, and even a certain amount of guesswork.
Many researchers have recognized these problems and have investigated
guidelines and models for the perception of information graphics.1 Much
work has been done on the efficacy of different visual encodings,2,3
resulting in useful rules about the use of color, position, area, etc. to
represent different types of variables. Others, for example Healy et al.4 have
investigated how models of preattentive processing can be used in
designing visualizations.
But these lines of research do not address a key element in the efficacy of
an information graphic: the degree to which its perceptual organization
reflects the organization of the underlying data. Many authors have
stressed that to design successful information graphics, one must take into
account the effects of perceptual grouping. For instance, the work of
Kosslyn5 contains many examples in which unintentional grouping effects
lead to confusing displays. It would therefore be useful to have a tool that
helped designers assess the perceptual organization of their designs.
Some attempts have been made to model perceptual organization in
Received: 24 November 2003 information graphics. Tufte provides general guidelines, such as the
Revised: 30 January 2004 Macro/Micro principle.6 But quantitative models suitable for software
Accepted: 8 March 2004 implementation are rare. Several authors have analyzed special classes of
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
124

displays: Tullis7 analyzes alphanumeric screens; Schnei-

derman et al.8 investigate standard Visual Basic dialog
boxes. The work of Saund9 on deriving perceptual
structure in the context of sketch editing is more
ambitious, but still requires a vectorized version of a
graphic as input. Because it is not amenable to the
analysis of non-vector-based visualizations, it is proble-
matic to apply this method to the output of existing
programs. (Portions reprinted, with permission from
Martin Wattenberg10 (c) 2003 IEEE.)
In this paper, an extension of the work by Wattenberg
and Fisher10 we introduce a formal model of visual
organization which can be applied to a broad class of
information graphics. We present an algorithm that takes
as input an arbitrary grayscale image, and returns as
output an analysis of the images organization that links
perceived structures at different scales. We do not claim
that this technique captures all or even most of the
aspects of human visual perception that would be far
Figure 1 Visual hierarchy, hand-drawn, for a piece of text. (The
beyond any current system but we do propose it as a
Dr. Seuss image.)
potentially helpful new model of a particular aspect of
perceptual organization.
We then describe a prototype software tool that applies
this model to help designers see how an information upside down to get a better sense of its organization. In
graphic may be understood by viewers. We demonstrate many ways, it would be helpful to have a mathematical
the utility of the model by exhibiting a variety of model that matched the standard designers intuition.
examples in which it captures aspects of design aes- Such a model could be useful to designers, for instance,
thetics; we also show how it can be used in the redesign who could apply it to early designs to see if the structure
of a real-life visualization. Finally, we discuss directions matched what they wished to communicate. It could also
for validating and extending the model. be helpful in automating some aspects of design for
instance, a computer might try to use the model to
optimize the correspondence between visual structure
Multi-scale model of visual organization
and data structure. All of these potential uses rely on a
Motivation: importance of multiple scales precise model that can implemented algorithmically.
Most information graphics display structure at several
different scales. That is, an image will contain large-scale Human and machine vision Psychologists have long
organization as well as many smaller details. Our studied perceptual organization and its multi-scale
hypothesis is that at all these scales the visual structure aspects. A full review of the psychological literature on
should reflect the structure of the data being conveyed, this topic is beyond the scope of this paper, but we cite a
with large-scale organization reflecting a broad overview few reference points. Gestalt psychologists, starting with
or summary, and smaller details reflecting details of the Wertheimer,12 have proposed a number of laws for how
data. As Bertin11 puts it: the brain groups objects: by proximity, good continua-
tion, and so on. Multi-scale aspects of grouping have also
A graphic should not show only the leaves; it should show been addressed in several lines of research.13,14 Many of
the branches as well as the entire tree. The eye can then go
these theories of grouping were qualitative, but investi-
from detail to totality and discover at once the general
gators have worked on creating quantitative or algorith-
structure and any exceptions to it.
mic models as well. Typical examples from this large
This intuition about multiple scales is shared by many research area are Kubovy,15 who treats grouping by
visual designers. Typographers, for example, routinely proximity in dot lattices, and the work of Li16 on neural
speak of a visual hierarchy in text layouts. Figure 1 shows network simulations of cortical processing. One interest-
a hand-drawn example of such a hierarchy. (We have ing system is Logans CODE theory of visual attention,17
chosen a piece of text as an example for analysis in the which has in fact been applied to information visualiza-
next section since it has several natural, unambiguous tions.18 Like many other psychologically derived models,
scales: letter, word, line, and paragraph.) however, this system requires feature constructs as input
Despite the general belief that multi-scale structure rather than direct pixel data, meaning that they cannot
exists and is important, that structure can prove surpris- be directly applied to images of information graphics.
ingly elusive. Experienced designers use techniques such The field of machine vision, however, provides a
as looking at an image from across a room or holding it different and more immediately fruitful perspective,

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
125

and is a rich source of pixel-level models. Analyzing

visual structure has long been recognized as an important
component of computer vision (see Witkin and Tenen-
baum19), and modern computer vision frameworks
typically are designed to be applied to arbitrary images.
In this paper, we highlight one particular framework,
scale space theory, and through a series of examples
suggest that it is particularly suitable for the analysis of
Figure 2 fs for the Dr. Seuss image from Figure 1, where s 8,
information graphics. A natural future direction would be
16, 44.
to reconnect this model with psychological work through
experimental validation.
1 x2 y2 =2s2
Gs x; y e :
Limits and assumptions 2ps2
Rather than attempting to model the full range of visual
experience, we focus on non-interactive motionless We define then fs by
grayscale images, and make no attempt to reconstruct a f s f
Gs ;
three-dimensional (3D) scene. By eliminating from
where * represents convolution. Informally, the function
consideration color, depth, motion, and interactivity,
fs represents the original image having been blurred by a
we simplify the domain considerably yet retain signifi-
factor of s. Figure 2 shows fs for three different values of s,
cant generality, for example encompassing a significant
using the same starting data as the top-left of Figure 1.
fraction of printed information graphics. Furthermore,
The 3Ds pace formed by the spatial dimensions x,y and
even within the domain of static grayscale images, we do
the new scale dimension s is known as scale space, and by
not attempt to create a complete model of visual
analyzing the functions fs on this 3D scale space we can
grouping. Instead, as a first step, we focus on a single
uncover important structures in the original two-dimen-
type of structure. Obviously, it would be desirable to have
sional (2D) image. We refer to s as the scale of the
a model that eventually did account for the many other
image, and so describe Figure 2 as being seen at three
dimensions of visual perception, and in the final section
different scales.
we discuss potential generalizations.

Structure and segmentation Having defined scale space

Our model: mathematical definition
we now need a notion of structure or organization at a
We now define our model. First we make precise the idea
given scale. There are many possible ways to define a
of scale. Then we define a simple method of extracting
structure. We choose to define structure by creating a
structure at a given scale. Finally, we describe a technique
segmentation of the image at each scale. For a given scale
for linking structures found at different scales. Through-
s, we follow Marr and Hildreth24 and consider the
out, we illustrate our progress through the algorithm
difference-of-Gaussians edge detection function
with the data in the top-left of Figure 1.
gs fs  f3s=2 :

Scale space We base our model on the classical machine

vision concept of scale space. Scale space theory1922 is a This function is one of the best-studied edge detectors,
formalism that describes the structure of a signal at many and has some correspondence to the responses of retinal
neurons.24 It is a close approximation of another classical
different scales at once. (Despite the similar name and
notation, scale space in this sense is not directly related to edge detector, the Laplacian operator, but numerically
the space-scale diagrams of Furnas and Bederson,23 an more stable. Figure 3 shows the function gs for the Dr.
elegant application of Riemannian geometry to zooming Seuss image at three different scales.
user interface design.) We can then naturally segment the square as follows.
To define scale space precisely, we need some notation. Consider the two (disjoint) regions where gso0 and gs40.
First, we represent the input image as a function: We let the connected components of these regions form
the elements of our segmentation. The sign of gs is thus
f : 0; L0; L ! 0; 1: constant in a given segment and has some significance; it
corresponds, very roughly, to whether the segment is
That is, we take f to be a function on a square of side L, brighter or darker than its neighbors.
where a value of 0 corresponds to black, 1 to white, and Why use the difference-of-Gaussians edge detector? It
values in between correspond to shades of gray. has several advantages. First, simplicity: it is well-under-
Given the function f, we then extend its domain to a stood and efficient to calculate. Second, unlike several
3D scale space by a special family of functions fs, where other popular edge detectors,25,26 the difference-of-
sZ0. First, let Gs be a Gaussian kernel with width s; more Gaussians method has the benefit of immediately produ-
formally, let cing closed contours, thus creating a segmentation

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
126

without additional steps. Third, the sign of the function but interesting visualization issue: informal tests showed
gs is useful in creating an algorithmic version of the that for complex segmentations, users found these scale
linking step below. Despite these advantages, it is space cartoons easier to interpret than the outline view
important to note some well-known drawbacks to this commonly seen in computer vision output. Note how
technique: poor localization, rounded corners, and over- closely the images in the bottom row match the hand-
sensitivity.27 A different edge detector would not, how- drawn diagrams of Figure 1.
ever, fundamentally alter the framework of our model. The scale space cartoons do raise some new issues,
Figure 4 shows the resulting segmentation at scales of however. One potential concern is that two adjacent
8, 16, and 44. In the top row the edges of segments are segments with similar average values may be difficult to
shown. In the bottom row, each segment has been filled distinguish. In many cases, however, this difficulty
with a single gray tone representing the average grayscale simply reflects the fact that the visual difference between
value of the pixels in the segment, a technique we call a the two segments is relatively unimportant. In situations
scale space cartoon. The scale space cartoon itself is a small where drawing attention strongly to all segmentations is
necessary, one might draw a faint outline around each
segment.
Edge detection is not the only way to locate structure at
a given scale. Probably the most common method one
used in many of the original scale space papers is to
analyze local maxima and minima of the function fs.21,28
Often this analysis is accompanied by some sort of
watershed segmentation.22,29 We tried several variants
of this technique but found they produced poor results,
possibly due to the non-generic nature of typical
Figure 3 Difference of Gaussians: fsf3s/2 , s 8, 16, 44. 50% information graphics. Compared to images of natural
gray is zero; dark gray is negative; light gray is positive. scenes, diagrams and visualizations have an unusual
number of areas of nearly uniform brightness. In many
cases we found that fs contained ridges, valleys, and
plateaus that were almost but not quite level, leading to a
proliferation of local extrema that did not correspond to
useful features in the image. Figure 5 shows an example.
On the left is a simple graph. On the right are the
boundaries of regions found by watershed segmentation
for local minima (as in Lindberg22) for scale s 16. It is
clear that the graph line itself has dissolved into many
individual segments, because the smoothed function has
many almost indistinguishable extrema in the area of the
main graph line in the image. This effect, which in no
way reflects the visual experience of viewing the graph, is
why we chose the edge-detection scheme described
above.
Figure 4 Algorithmically derived segmentation of the Dr. Seuss
image for s 8, 16, 44. Top: edges of segments. Bottom: filled Linking structures at different scales As described so far,
segments, or Gestalt cartoons. the model finds structure only at a single scale. But the

Figure 5 An image (left), and its boundaries as found by watershed segmentation (center). Contrast with the results of our edge-
detection algorithm (right).

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
127

Figure 6 Four scales of Dr. Seuss.

perceptual structure of an image includes not just the

structure at one scale, but the relationships between
features at different scales. In the scale space literature,
linking features between scales is often referred to as
finding the deep structure of an image.21 In this section we
Figure 7 Linked segments in L at different scales for part of
describe a novel method of finding this deep structure
the Dr. Seuss image.
that is particularly useful for information graphics.
Consider the segmentations in Figure 6, shown as a
series of scale space cartoons. It is visually clear that the
two blobs in the s 11 view correspond to the individual
letters of the words Dr. and Seuss respectively. The final
part of our model is a method of making this intuition
precise.
Let S1 and S2 be two image segments found at scales
s1rs2 respectively. We can naturally view S1 and S2 as
embedded within the 3D scale space, that is, as the sets
{s1}  S1 and {s2}  S2. We will say S1 is linked to S2, denoted
by S1rS2, if either S1 S2 or there is a continuous curve C
through scale space from a point on S1 to a point on S2,
such that gs maintains the same sign on C and s is
monotonically increasing along C. It is easy to verify that
the relation r defines a partial order on the set of
segments. It is also clear from the definition that this
partially ordered set breaks into two disconnected
components, one that corresponds to the subset of
segments where gso0, which we denote as L and one
we call L where gs40. (It is possible for each of these
two sets to have many maximal elements.) In some cases,
L and L turn out to correspond to foreground and
background elements. For example, in the Dr. Seuss
image, the segments corresponding to the text are Figure 8 The linked structures in L for the entire Dr. Seuss
represented in L while the whitespace is represented in image.
L.
Figure 7 is a visualization of the results of connecting
linked segments in L for the Dr. Seuss image. abstract graph-theoretic views of the lattice and a layout
The image shows a 3D view of scale space, with four of 2D thumbnails with connections drawn between
separate planes highlighted (corresponding to s 1, 4, 7, segments. In these cases, however, users were uniformly
11). For each plane, we show the segmentation for the confused about the connection between the lattice
corresponding s value, and for each pair of linked structure and the image.
segments in adjacent planes we have drawn a line For completeness the L lattice for the entire Dr. Seuss
between the segments centroids. For simplicity, in this image is shown in Figure 8. Again, the structure nicely
diagram we only show L, the segments with negative gs, corresponds to the intuitive hierarchy of paragraphs,
since they account for the main visual structure. The lines, words, and letters.
result is a tree structure on the words that corresponds to Although linking structures at different scales by
the intuitive hierarchical division of a phrase into words following zero-crossings of various operators is common
and words into letters. in scale space theory,22 the particular linking described
The choice of a 3D display is a visualization exercise in here is unusual, and in fact is a key distinguishing feature
its own right. We tried various alternatives, such as of our model. Most scale space segmentation algorithms

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
128

Figure 9 Image whose structure is not tree-like. Left: original

image. Right: structure of L.

seek a hierarchical segmentation of an image, where the

partial order is always a tree structure. The segmentation
described above, however, can produce non-nested
segments with non-tree lattices. In the context of scene
segmentation and object recognition the conventional
applications of scale space theory this is an undesirable
property. But as several authors have pointed out,9,29 a
non-tree lattice seems to model well the visual experience
Figure 10 An image (top), with scale space cartoons at high
of certain images. Indeed, given that the goal of many
(middle) and low (bottom) resolution.
information graphics is to portray complex inter-relation-
ships, any model that led to pure trees would be of
limited applicability. leaf region can be subdivided to create a new tree branch
Figure 9 gives an example of an image whose visual of child nodes without disturbing the rest of the
structure is not tree-like. The barbell image, at a small structure.) Some complex non-tree structures are also
scale, is one continuous object, at a slightly larger scale realizable. For example, Figure 10 shows an image with 3
breaks into two main parts, and at a large scale merges leaf nodes that split into many more items, and then
into one object again. merge back into 3 parent nodes, each of which is linked
to one of the original leaves. This example indicates the
Possible structures Since the analysis technique de- variety of possible structures; classifying realizable struc-
scribed here produces visual structures that have the tures completely is an interesting problem for future
form of a lattice, a natural question is whether all finite research. The example also shows a limitation of our
lattice structures can be represented, or realized, by some method: the large-scale structure is not always the most
image. That is, how expressive is the system: Given a data salient to a human observer: most people would likely
set with an arbitrary lattice structure L, is there always an describe the original diagram as three horizontal objects
image whose visual structure is L? We do not know the rather than 3 vertical ones,30 although the vertical
answer, but conjecture that there are classes of lattices grouping is a notable feature of the graphic.
that are not realizable. Possibly this may be easier to
prove when the segmentation is performed according to Related methods The general concept behind our con-
the zeros of the Laplacian operator, which is similar to struction, analyzing a signal at multiple resolutions, is
the difference-of-Gaussians method used here but allows found in many fields. One closely related technique of
the introduction of mathematical machinery related to multi-scale analysis is the continuous wavelet transform.
the heat equation. The difference-of-Gaussian operator used in our segmen-
At the same time, it seems likely that many structures tation step is in fact a close approximation to the
are realizable. For example, all trees are realizable via a Mexican Hat wavelet.31 Statisticians use convolution
treemap-like diagram. (To see how this could work, with Gaussian kernels of varying radii in kernel density
consider only black and white images where the black estimation,32 a non-parametric estimation technique;
region is the disjoint union of closed connected sets Leung et al.29 have applied scale space theory to statistical
corresponding to leaves. We can then show that any clustering using a watershed-type segmentation techni-

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
129

que. A third technique that is closely related is the multi-

scale pyramid representation.33 Originally used for image
compression, it is interesting to note that this structure is
now used in at least one sophisticated model of visual
perception and attention.34

Results and applications of the model

To test our model, we built a software tool that applies
the model to arbitrary input images. The tool was used to
create all the images in this paper, with the exception of
the hand-drawn Figures 1, 13, and 14. As a demonstration
of our model, we apply it to three case studies, and show
how it can be used in the redesign of a real-life
visualization.

The software
The software tool contains the following numerical
approximation of the model. We represented the image
functions fs as 2D arrays of floating-point values (one per
pixel in the original image), and computed fs for only a
few discrete values of s. To perform linking, we looked at
each pair of successive approximations to fs, and connect Figure 11 Gestalt cartoons showing differentiation of figure
any two segments that share a sign and which overlap. and ground in a graph. Left: thin grid lines. Right: thick grid
Our implementation is written in Java, and on a 700 MHz lines.
Pentium 3 PC requires up to a minute to perform a full
structural analysis on a 800  600 pixel image at 15 scales.
Once the analysis is performed, it is saved for viewing as human can segment the graph in the second diagram by
both a series of grayscale images and as a 3D VRML file. using orientation information, which our model ignores.
We describe the interface of the application in more Nonetheless, doing so places an additional cognitive
detail in Section 4. burden on the viewer, and in fact it is a standard principle
of information design that grid lines should be signifi-
Questions of scale An important consideration for cantly lighter than lines representing foreground data.
computational algorithms is to ensure that they are able Thus the model indicates, correctly, that there is a
to handle large data sets. Because our system examines problem with the second graph. This situation where
the output of a visualization algorithm the pixels it a minor visual change has a large effect on comprehen-
produces and not the underlying data set, it is limited sibility is exactly where it is useful to have a model.
by the size of the screen. The segmentation and linking
steps each require time and memory proportional to the A famous real-life example
number of pixels, since each pixel is marked once in an How does the model fare on a real-life example? Figure 12
offscreen buffer. For a constant s, computing gs is also shows scale space cartoons for a complex scatterplot, the
linear in the number of pixels. Since the algorithm is not famous astronomical HertzprungRussell diagram. This
affected by the number of items in the underlying data scatterplot, which displays data on stars with temperature
set for a visualization, and indeed works by progressively on the x-axis and absolute magnitude on the y-axis, plays
simplification of an image, it is in practice computation- a central role in scientists conception of stellar evolu-
ally feasible. In addition, the examples below show that tion. The HR diagram at the top left of Figure 12 is
often only a small subset of scaling levels need to be reproduced directly from Spence and Garrison,35 which
examined in order to reveal useful information about contains a detailed discussion of this historically sig-
perceptual grouping. nificant information graphic.
The segmentations in the scale space cartoons capture
A simple example: graphs and grid lines the intuitive experience of reading the diagram: the
Our first example shows scale space cartoons of two small-scale (s 4) view emphasizes the vertical structures,
versions of a simple graph (Figure 11). At top left is a while at s 8 and 16 the large-scale clusters stand out.
graph with thin gridlines, at top right is a graph with The areas highlighted for s 16 correspond nicely to the
overpoweringly thick ones. The segmented versions at standard organization given by human experts: Figure 13
scale s 4 are shown below. In the graph with thick shows how an astronomer structures the diagram.
gridlines the graph itself is not segmented from the One subtlety is that at high scales some regions of the
background. This is an interesting indication of both the scale space cartoon become extremely faint, making
strength of our model and one of its limitations. A them difficult to see. In Figure 12, for example, the area

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
130

Figure 13 Human expert partitioning of HR diagram. After

Fix.36

Figure 12 Original image and gestalt cartoons of the Hertz-

prungRussell diagram. Reprinted with permission from The
American Statistician. Copyright 1993 by the American Statis-
tical Association. All rights reserved.

corresponding to the white dwarfs section toward the

bottom left has several subtle regions, barely visible at Figure 14 Left: sketch of portion of Market Map. Right: A
s 8. This problem could be fixed by outlining the redesign with stronger borders.
regions in a dark color, but that tactic could bring its own
problems, implying stronger salience for small regions
than is actually the case.
The regions labeled A and B in Figure 10 show another redesigned version with darker and thicker borders. (See
example of how a non-tree structure can be an appro- Figure 14).
priate model. The region labeled A is a single large When we fed these images into our model, the results
segment, reflecting the small-scale structure of a com- were clear. Figure 15 shows the structure derived for the
bined dense vertical and diagonal cluster. But a larger current version. Note that the lattice structure is
scale, s 8, that segment has broken into two parts, at B, complex, confusing, and does not follow the underlying
corresponding to the giants and main sequence regions hierarchy of the data items. At point A in the diagram, for
in Figure 13. Since at the highest scale these segments example, two items in different groups are spuriously
merge again, we see that the model produces a non-tree joined. In Figure 16, the lattice structure is far simpler
lattice structure. This contrasts with many clustering and close to a perfect tree. The key is that the borders
methods and with conventional scale-space segmenta- demarcating regions are now significantly darker than
tion techniques, which produce trees only. the filled rectangles, and that widths of the borders at
different levels of hierarchy are much different. This
dramatic result has led to a reconsideration of the
A treemap redesign original design exactly what we would want from a
Finally, we discuss how the model can inform the design perceptual model.
of a visualization. We take as our example the SmartMo-
ney Market Map37 a treemap visualization38 that displays
data on several hundred publicly traded stocks. The first The scale space cartoonist application
author of this paper, who led the design of the Market In this section, we provide a more detailed description of
Map, has on many occasions heard the comment that the the end-user application for applying our scale space
borders between regions are not strong enough. His model, currently named The Scale Space Cartoonist. The
intuition, however, was always that they were perfectly application is written in Java and is platform-neutral. We
fine as is. Since this is exactly the kind of design issue will first describe the graphical pieces that make up the
where a perceptual model would be useful, we decided to application. We believe that the general framework may
apply our software tool. To make a comparison, we be useful in for other applications aimed at automatically
created a stylized version of the current Market Map and a analyzing graphics.

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
131

Figure 17 The Scale Space Cartoonist interface.

Figure 15 L structure of original map design at scales up to

s 20. Some flaws: A, two items in different groups are scales. In informal tests, the previews proved to be
spuriously joined; B and C, a single group is spuriously critical, allowing users to quickly view cartoons of the
separated. image at a variety of scales. Each thumbnail is clickable so
that the user may easily select one for closer examination.
As mentioned above, a full structural analysis may take
up to a minute for a full-screen image, but the small size
of the thumbnails, and the fact that they represent only a
small subset of scale, means they can be generated in
under a second.
The menus at top allow the user to customize the view
and to perform simple input and output. Users may
analyze an image either by loading a file or pasting an
image from the clipboard. (The latter feature allows easy
analysis of running programs: after capturing a screen
dump, a user can simply paste the clipboard image into
the Scale Space Cartoonist.) The Filter menu allows the
user to change between different visual filters. The
default filter is the difference-of-gaussians segmentation
described in the section on our model mathematical
definition, but the program also enables viewing of
simple Gaussian blurs and a Laplacian convolution. The
Display menu allows the user a choice between different
views of the processed image. The default is a scale space
cartoon, but the user can also view raw image data as in
Figures 24. One option that seemed initially promising
but proved confusing for viewers was to use alpha
blending to overlay a scale space cartoon (in a shade of
Figure 16 L structure of redesigned map. Grouping is almost orange) on top of the original black and white image.
perfect; only flaw is an orphan item at A. Although we had hoped that this would provide a good
way for users to understand the connection between the
cartoon and the image, it was hopelessly confusing in
Interface components practice.
The application is composed from four main components The application also allows two special viewing modes.
along with an auxiliary module for 3D viewing (see In some cases, it is helpful to see the scale space cartoons
Figure 17). The large panel at the right displays a cartoon for many scales simultaneously. A menu option lets the
of the current graphic. On the left is a pane with user choose to make a contact sheet image, showing a
thumbnail views of the analysis of the image at different packed array of cartoons at any set of scales desired. Such

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
132

contact sheets are convenient for printing. A second a completely different type of data. Consider a line graph.
menu option lets the user create a 3D VRML view of the As we showed above, our scale space model can provide
scale space analysis, viewable in a web browser with a insight into whether the graph stands out from a
VRML 2.0 plug-in (e.g. Parallelgraphics Cortona plug-in). background grid. The model does not, however, provide
The 3D view is good for detailed analysis and demonstra- any insight into the numerical values conveyed by that
tions, but is difficult to reproduce clearly in print. graph which is clearly the most important data it
conveys. It is therefore unlikely that scale space analysis,
Current limitations and future directions no matter how it is extended, will become the sole tool
We have a described a model that is by no means for evaluating the perception of information graphics.
complete. Here we discuss various limitations and Finally, although the scale space method may help
possibilities for addressing those limitations. Perhaps identify flaws in presentation, it currently does not give
most important, the model is at its core a psychological suggestions for fixing those flaws. A potentially fruitful
hypothesis and therefore cries out for experimental area of long-term investigation would be using the model
validation. There are several natural directions to in- to automatically optimize information graphics. That is,
vestigate. One tactic would be to compare the structures given a known data structure one could attempt to find a
generated by our model with self-reports of users method for displaying that structure in an optimal
perceptions. A more pragmatic validation would be to manner according to the model detailed here. This could
study whether, in using the software tool described here, ultimately involve either an algorithm that incrementally
creators of information graphics are able to modify their improved a representation, to find a local optimum, or
designs in a way that user studies show are beneficial. even some method of mapping a structure directly to a
Two obvious shortcomings of our model are that it globally optimal visual representation.
applies only to grayscale images and that it addresses
only one type of grouping mechanism. One of the
reasons to choose scale space analysis as the basis for Conclusion
our method is that there is a rich body of research We proposed a new technique for modeling multi-scale
extending the basic idea to more general aspects of image perceptual organization in information graphics. The
structure. Theories that handle color or orientation have model is based on a classical machine vision technique,
been proposed14,34,40,41,43 and could be applied to our scale space, with a novel method of creating links
model. In addition, increasingly sophisticated visual between structures at different scales. We demonstrated
perception models, based on psychology, are becoming how a software implementation of this model captures
increasingly available.30,39 Some of those models fruit- important aspects of design aesthetics for several infor-
fully address notions of connectivity and suggest a broad mation graphics, and gave an example of how it may be
set of ways that humans cluster information. Orienta- used to give input into questions of design. We believe
tion-sensitive models have the potential to address the there is sufficient evidence of promise that it is worth
fact that our method often confers insufficient saliency extending and validating the model.
on lines and curves, which can lead to unsatisfactory
analyses for graphics such as node-and-link diagrams. It Acknowledgments
may also be advantageous to use a more sophisticated Thanks to Ben Shneiderman for helpful comments on the
segmentation method than the difference-of-Gaussians purpose of modeling perception, and to Dan Gruen for
edge detection employed here, since in some complicated advice on cognitive psychology. Thanks to Kushal Dave,
images the simple segmentation algorithm described here Steve Rohall and Bernard Kerr for comments on this
can yield counterintuitive results. It would also be useful manuscript, and to the IBM Research CUE group for support
to investigate ways of optimizing the numerical algo- and encouragement. We also acknowledge the anonymous
rithm to run in an interactive timeframe. referees for several helpful suggestions and references.
A deeper limitation is that the model relies on a Figure 12 reprinted with permission from The American
directed graph to describe the perception of an image. Statistician. Copyright 1993 by the American Statistical
Obviously, in many cases an information graphic conveys Association. All rights reserved.

References
1 Ware C. Information Visualization: Perception for Design. Morgan- 4 Healey CG, Booth KS, Enns JT. Harnessing preattentive processes for
Kaufmann: San Francisco, 2000. multivariate data visualization. Proceedings Graphics Interface 93,
2 Cleveland W. The Elements of Graphing Data. Wadsworth: Monterey, 1993; 107117.
CA, 1980. 5 Kosslyn SM. Understanding charts and graphs. Applied Cognitive
3 Mackinlay JD. Applying a theory of graphical presentation to the Psychology, 1989; 3: 185226.
graphic design of user interfaces. Proceedings of the ACM SIGGRAPH 6 Tufte E. Envisioning Information. Graphics Press Cheshire, Connecti-
Symposium on User Interface Software (UIST 88), 1988; 179189. cut, 1990.

Information Visualization
 Analyzing perceptual organization in information graphics Martin Wattenberg and Danyel Fisher
133

7 Tullis TS. A computer-based tool for evaluating alphanumeric 27 Parker JR. Algorithms for Image Processing and Computer Vision. John
displays. Proceedings of INTERACT84, 1984. Wiley & Sons: New York, 1997.
8 Shneiderman B, Chimera R, Jog N, Stimart R, White D. Evaluating 28 Witkin A. 1983. Scale-space filtering. Proceedings of the Eighth
spatial and textual style of displays. Proceedings of Getting the Best International Joint Conference on Artificial Intelligence (IJCAI 83).
from State-of-the-Art Display Systems 95. London 1995. Karlsruhe 812 August, 1983..
9 Saund E. Symbolic construction of a 2-D scale-space image. IEEE 29 Leung Y, Zhang JS, Xu ZB. Clustering by scale-space filtering. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 1990; 12: Transactions on Pattern Analysis and Machine Intelligence, 2000; 22:
817830. 12.
10 Wattenberg M, Fisher D. A multi-scale model of perceptual 30 Palmer S, Rock I. Rethinking perceptual organization: the role of
organization in information graphics. InfoVis 2003, 2003. uniform connectedness. Psychonomic Bulletin and Review, 1994; 1:
11 Bertin J. The Semiology of Graphics. University of Wisconsin Press: 2955.
Madison, WI, 1983. 31 Antoine JP, Carrette P, Murenzi R, Piette B. Image analysis with two-
12 Wertheimer M. Gestalt theory. In: Ellis WD (Ed). A Sourcebook of Gestalt dimensional continuous wavelet transform. Signal Processing, 1993;
Psychology. The Humanities Press: New York, 1924/1950; 1110. 31: 241272.
13 Navon D. Forest before trees: the precedence of global features in 32 Scott DW. Multivariate Density Estimation. Wiley-Interscience: New
visual perception. Cognitive Psychology, 1977; 9: 353383. York, 1992.
14 Palmer SE. Hierarchical structure in perceptual representation. 33 Burt PJ, Adelson EH. The Laplacian pyramid as a compact
Cognitive Psychology, 1977; 9: 441474. image code. IEEE Transactions on Communications, 1983; 31:
15 Kubovy M. The perceptual organization of dot lattices. Psychonomic 532540.
Bulletin & Review, 1994; 1: 182190. 34 Itti L, Koch C. Computational modeling of visual attention. Nature
16 Li Z. A neural model of contour integration in the primary visual Reviews Neuroscience, 2001; 2: 194203.
cortex. Neural Computation, 1998; 10: 903940. 35 Spence I, Garrison RF. A remarkable scatterplot. The American
17 Logan GD. The CODE theory of visual attention. Psychological Review, Statistician, 1993; 47: 1219.
1996; 103: 603649. 36 Fix JD. Astronomy: Journey to the Cosmic Frontier. (Second edition,
18 Pirolli P, Card SK, van der Wege MM. Visual information foraging in a updated) McGraw-Hill: New York, 1999.
focus+context visualization. Proceedings, CHI01, Human Factors in 37 Wattenberg M. Visualizing the Stock Market. CHI99 Extended
Computing Systems, 2001. Abstracts, 1999.
19 Witkin A, Tenenbaum M. On the role of structure in vision. In: 38 Shneiderman B. Tree visualization with tree-maps: 2-d space-
Rosenfeld A (Ed). Human and Machine Vision. Academic Press: New filling approach. ACM Transactions on, 1992; 11: 9297.
York, 1983. 39 Geisler WS, Super BJ. Perceptual organization of two-dimensional
20 Iijima T. Basic theory of pattern observation. Papers of Technical Group patterns. Psychological Review, 2000; 107: 677708.
on Automata Control, IECE, Japan, 1959. 40 ter Haar Romeny B, Geusebroek JM, Van Osta P, van den Boomgaard
21 Koenderink JJ. The structure of images. Biological Cybernetics, 1984; R, Koenderink JJ. Color differential structure. Scale-Space 2001, 2001;
50: 363370. 353361.
22 Lindeberg T. Scale-Space Theory in Computer Vision. Kluwer: 41 Kalitzin B, ter Haar Romeny B, Viergever M. Invertible orientation
Dordrecht, 1994. bundles on 2d scalar images. Scale-Space Theories in Computer Vision.
23 Furnas GW, Bederson B. Space-scale diagrams: understanding Springer: Berlin, 1997; 7788.
multiscale interfaces. Proceedings CHI 95. Human Factors in Computing 42 Mozer M, Zemel RS, Behrmann M, Williams CKI. Learning to
Systems, 1995. segment images using dynamic feature binding. Neural Computation,
24 Marr D, Hildreth E. Theory of edge detection. Proceedings of the Royal 1992; 4: 650665.
Society of London, Series B, 1980; 207: 187217. 43 Perona P, Malik J. Scale-space and edge detection using anisotropic
25 Canny J. A computational approach to edge detection. IEEE diffusion. IEEE Transactions on Pattern Analysis and Machine Intelli-
Transactions on Pattern Analysis and Machine Intelligence, 1986; 8: gence, 1990; 12: 629639.
679698. 44 Wickens CD, Carswell CM. The proximity compatibility principle: its
26 Shen J, Castan S. An optimal linear operator for step edge detection. psychological foundation and relevance to display design. Human
Computer Vision, Graphics, and Image Processing, 1992; 34: 321343. Factors, 1995; 37: 473495.

Information Visualization
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.