Mapping the Uncharted
What can GenAI
(Really) do for
data visualization?
Video autoplay is off
Certain Gen-AI tools, by allowing us to focus on our *intention* within
visualizing data, can help us break free from the confining framework in
which traditional mapping tools need to fit and restrict data.
This text is based in part of the thoughts and reflections form the CfD
Conversation Series in the Northeastern University with Paolo Ciuccarelli,
Enrico Bertini, Vidya Setlur: (Senior Director, Tableau Research) and Victor
Dibia (Principal Research Software Engineer, Human-AI eXperiences (HAX)
Team, Microsoft Research)
"To understand something is to understand its topography, to know how
to map it. And to know how to get lost in it." — Under the Sign of
Saturn, (Sontag, 1980).
Data floods more and more into every corner of our existence. Yet, as
Susan Sontag eloquently suggests, true comprehension arises not just
from mapping known terrains, but also from venturing and getting a bit
lost into uncharted territories. Therefore, to fully understand data,
visualizing it according to conventional methods might not always be
enough. Instead, sometimes it is necessary to sail adrift in the great sea
of data, in order to take a step back and recognize the most important
motions and processes to address, looking at it with a fresh pair of eyes.
Image generated using krea.ai’s Flux 1.1 Pro Ultra model
Today, I want to make a case for the potential of using Gen-AI tools to
help data visualization bridge quantitative data with qualitative human
experiences. Certain Gen-AI tools, by allowing us to focus on
our *intention* within visualizing data, can help us break free from the
confining framework in which traditional mapping tools need to fit and
restrict data. This can help shift the focus towards the human portion of
quantitative data, rather than mere numbers and statistics.
DISCLAIMER: I’ve spent a big part of the last two years researching and
demystifying Gen-AI all over, from pointing out the ethical problems of
non-consented use of images and texts for LLM production to the
entangled biases and the dangers of hallucinations in mainstream tools
such as Chat-GPT — but it’s too late to be just pessimistic, so this is one of
a series of articles trying to find the right ways to use these tools.
READING DATA THROUGH A NEW LENS
We often interpret data under the influence of what might be termed as
the “Cartesian anesthetic gas” ¹ — which we adapt here to describe a
metaphorical haze that reduces complex realities to mere numbers and
graphs. This reductionist lens, inherited from Cartesian dualism,
encourages a separation between the quantitative and the qualitative,
the measurable and the experiential. As a result, data is often stripped of
its cultural, emotional, and humanistic contexts, limiting our
understanding and connection from the reality that this data depicts.
In many instances, this approach leads to a myopic view where data
becomes an abstract entity, detached from the very realities it aims to
represent. At DDS, an important part of our work is to recognize that
data simultaneously sustains societies and perpetuates systemic
structures. This shows the ability of data to both aggravate inequalities
and offer tools to reduce them.
Image generated using krea.ai’s Flux 1.1 Pro Ultra model
Surveillance capitalism has transformed data into a raw material,
extracted and distilled through the millions of physical and digital
interactions worldwide. Companies collect vast amounts of personal
information, turning our behaviors and preferences into commodities.
We often consume this data through dashboards, infographics, charts,
and other visualization systems that, while efficient, sometimes fall short
in depicting the full spectrum of realities they attempt to illustrate. These
tools can oversimplify complex social phenomena, leading to
misunderstandings or superficial analyses. The map, no matter how
detailed, is always a reduction of the territory. Similarly, the “like” button
on social media platforms can never encapsulate the wide spectrum of
human experiences associated with genuinely liking or appreciating
something. It reduces nuanced emotions to a single, quantifiable action,
erasing the depth behind our interactions.
To truly understand and engage with data, we need to explore beyond
these constraints. This exploration requires developing a multitude of
strategies — alternative, dissenting, and interdisciplinary — that
challenge conventional methodologies. By incorporating insights from
academia, public and private organizations, and using technologies such
as Gen-AI, we can try to look beyond these limitations of data.
Empowering users through adaptive
tools/Command vs. Intent
One of Gen-AI’s most promising aspects is its ability to facilitate the
development of tools that adapt to our thinking patterns, rather than
force us to conform to rigid interfaces. In my case, this has transformed
how I research, analyze, and create, turning it into a more intuitive and
personal experience.
Traditional data visualization tools have long been rooted in a command-
driven paradigm. Software applications like Microsoft Excel, Tableau, and
programming libraries such as Matplotlib or D3.js require users to specify
exact instructions to produce visual representations of data.¹ This process
involves meticulous attention to detail: selecting chart types, defining
axes, choosing color schemes, and adjusting numerous parameters to
achieve the desired outcome. Users must know precisely what they want
and how to command the tool to create it.
This emphasis on specificity and control can be both empowering and
limiting. On one hand, it allows for precise customization and fine-
tuning. On the other hand, it demands a steep learning curve and can
constrain creativity by forcing users to think within the confines of the
tool’s capabilities. Users must often focus more on the “what” can be
technically depicted rather than on the “why” certain visualizations would
be meaningful. This reduces data visualization to a mechanical exercise,
rather than an explorative process where one can step back, get lost in
the data, and organically uncover meaningful patterns and stories.
Interfaces like the ones developed by OpenAI or Anthropic, herald a shift
from command to intent. Unlike traditional tools, these AI systems
invite users to express their goals and motivations in natural language.
Instead of dictating specific commands, users articulate their intentions,
and the AI interprets these to generate the most probable outputs. The
interaction becomes more about the why — the underlying reasons and
desired outcomes — than the granular details of what needs to be done.
For example, a user might engage with GPT-4 by stating: “I want to
visualize the impact of climate change on coastal cities over the next 50
years, highlighting areas most at risk due to rising sea levels.” The AI
processes this intent, accesses relevant data, and generates a
visualization that aligns with these goals.
This intent-driven approach offers several advantages:
1. Focus on Purpose: By prioritizing the “why,” users can concentrate on
the insights they wish to gain and the stories they want to tell through
data.
2. Accessibility and Inclusivity: Users without extensive technical
expertise can participate in data visualization. Natural language
interfaces lower barriers to entry, democratizing access to powerful
analytical tools.
3. Enhanced Creativity: AI can offer new solutions and unexpected
representations which users might not have thought of when using
traditional tools. This can lead to new perspectives through which to
look at old data.
However, this shift also introduces some challenges:
1. Interpretation Ambiguities: AI models still misinterpret user intent,
especially if instructions are vague or ambiguous.
2. Reduced Control Over Specifics: Users seeking precise control over
every aspect of the visualization may find the AI’s abstractions
limiting.
3. Reliance on AI’s Understanding: The AI’s outputs are only as good
as its training data and algorithms. Biases or gaps in the AI’s
knowledge continue to affect the quality and accuracy of the
visualizations produced.
The transition from command to intent represents a broader evolution in
human-computer interaction. It mirrors the way humans naturally
communicate and collaborate, focusing on shared goals rather than
prescriptive instructions. This paradigm will shift a lot of data designers
to think more holistically about their data and the messages they wish to
convey.
Divergence and Convergence thinking
Every creative endeavor traverses two fundamental phases: divergence
and convergence. During divergence, we open ourselves to new ideas,
gather inspiration, and let our minds roam free. It’s a process of
exploration and expansion — a “breathing in” of possibilities.
Convergence, on the other hand, involves narrowing down options,
refining ideas, and perfecting details — a “breathing out” of precision.
Most traditional creative tools — like Adobe Illustrator, Figma, or Excel —
are designed for convergence. They excel in helping us polish and
perfect, but they often fall short in fostering the exploratory phase of
creativity. They are built to help us be precise, not to inspire unexpected
ideas.
Gen-AI tools, such as NLPs like Claude or GPT4, or generative models like
Stable Diffusion, disrupt this paradigm. They are inherently designed for
divergence, sometimes generating outputs that can spark new directions
in thinking. They can allow us to “breathe in” a vast array of possibilities,
enhancing the creative process. However, these tools can also fall into
the trap of offering a single, overly confident perspective, potentially
limiting our view of the world. For example, GPT-4 might present a
polished but narrow narrative based on the data it has been trained on,
while tools like Perplexity, which aggregate multiple sources and
perspectives, can help us maintain a broader lens, offering context and
nuance that foster true exploration.
***
The perils of overconfidence in algorithms
This captures a key paradigm: Gen-AI offers an immense potential when
it comes to bringing in new ideas and broadening the range of
affordances, making it an invaluable, positive tool for divergence. And
yet, it also presents significant challenges, which show that at the hour of
discernment and decision-making (the convergence phase), it’s crucial to
maintain the human component of critical thinking and holistic
understanding.
Algorithms often deliver outputs with an air of absolute confidence,
regardless of their accuracy. Unlike humans, machines lack subtle cues of
uncertainty — hesitations, tonal shifts, or body language — that help us
gauge the reliability of the information we are handed. That’s why, in
most of the experiments we have been working on in the studio, we try
to use Gen-AI as an adversarial agent: instead of solving the problem, we
put it at the service of finding flaws to our hypotheses and insights,
based on the data we have.
EXPLORING TEXT TO IMAGE VISUALIZATION
As the use of Gen-AI enables a wealth of new visualization languages
and possibilities, I’ve been pondering whether we can visualize data in a
way that not only conveys information, but also connects with the viewer
on a human level. This question has guided our exploration into
Generative AI and data visualization over the past year, to engage both
our minds and emotions.
Here’s a link to our promptbook — a guide created for scientists,
designers, journalists and storytellers, to inspire them to look through the
lenses of data language when using text-to-image software.
Excerpts from “Prompt book for data lovers II”
Streamers
In 2022, we started experimenting with early text-to-image models to see
how over 50 different semantic fields, enriched with more than 400
adjectives, could enhance data visualization. We drew inspiration from
Jacques Bertin’s semiology studies², which gave us foundational variables
like brightness, height, width, depth, and transparency. Bertin provided
the building blocks; we just decided to see how far we could stretch
them.
Excerpts from “Prompt book for data lovers II”
Streamers
Building Your Own Semantic Family
In our “Prompt Book,” we explain how to build your own set of visual
variables that match your data and message. It’s like assembling your
own Lego set for data visualization. You’re not just following instructions;
you’re creating something unique that speaks to your perspective.
Excerpts from “Prompt book for data lovers II”, @Domestic Data
Streamers
A Living Map: Visualizing Pollution Through
Moss
One of our projects visualized real-time pollution data by turning it into
images of moss-shaped countries. Instead of a static map, you got a
living organism that grows or withers based on environmental health.
Seeing this visualization was like walking through a garden where each
plant showed how well it’s cared for. It hit home more than any bar chart
could. It was a striking reminder to take better care of our planet before
all our moss — both real and virtual — calls it quits.
We’ve also developed systems that generate multiple visualizations
quickly. These aren’t polished masterpieces, but that’s kind of the point.
In about two hours, using tools like Midjourney and Google Slides, we
produced data representations that, while imperfect, showed the
potential of these tools.
Shown below are a few images of these quick drafts. They let us explore
our ideas without overthinking: often, the best insights come when
you’re not trying too hard — happy accidents do happen.
Explore more here.
BEYOND WORDS: THE ENDURING POWER OF DATA VISUALIZATION
All of this brings me to a question that Paolo Ciuccarelli asked during
one of our conversations — a question that left me pondering: “Is
visualization still relevant when we have much easier ways to extract
insights from data through conversational systems like ChatGPT?”
It’s a fair point. With AI models that can digest and regurgitate data
insights in plain language, one might wonder if visualizations are
becoming obsolete. But here’s my take: absolutely not. Visualization is
still a thing, and will be so for the foreseeable future.
Visualization isn’t just about representing data; it’s about offering a
perspective that might be impossible to grasp through words alone.
Think of it this way: trying to understand complex data solely through
text is like trying to appreciate a painting by reading a written description
of it. Sure, you get the gist, but you miss the nuances — the colors, the
brushstrokes, the emotions it evokes.
Visualizations provide a different language, one that can reveal patterns
and insights hidden beneath layers of numbers and statistics. They
engage our visual cognition, allowing us to see relationships and
anomalies that might elude textual analysis. It’s not just about making
data look pretty (though that’s a nice bonus); it’s about making data
accessible and meaningful on a human level.
We need more languages to understand reality than just the symbolic
ones. Words and numbers have their place, but they can’t capture
everything. Sometimes, a single image can convey what pages of text
cannot. Visualizations offer that alternative language — a way to bridge
the gap between complex data and human intuition.
In a world increasingly saturated with information, having multiple ways
to interpret and understand data isn’t just helpful — it’s essential.
Visualizations can simplify the complex, highlight the overlooked, and
make the abstract tangible. They invite exploration and can even spark
joy — or at least a satisfied nod of understanding.
So yes, even with conversational AI tools at our fingertips, visualization
remains a vital tool in our arsenal. It complements these technologies by
providing a different angle of approach, enriching our understanding
rather than replacing existing methods.
References:
¹ Hakim Bey. T.A.Z.: The Temporary Autonomous Zone, Ontological
Anarchy, Poetic Terrorism. Autonomedia, 1991. The term “Cartesian
anesthetic gas” is used to convey a subtle indifference to lyricism and
human subjectivity.
² Bertin, J. (1983). Semiology of Graphics: Diagrams, Networks, Maps (W. J.
Berg, Trans.). University of Wisconsin Press.
³ Jansen, Y. (2014). Physical and Tangible Information
Visualization (Doctoral dissertation, Université Paris-Sud XI).
