the articulation of

visual experiences
through algorithm
Carlos Lunetta
The articulation of visual experiences through algorithm
by Carlos Lunetta

Thesis submitted in partial fulfillment of the requirements

for the degree of Master of Fine Arts and is approved by the
MFA Design Review Board of the Massachusetts College of
Art in Boston.

May 2005

Jan Kubasiewicz
Thesis Advisor

Joe Quackenbush
Thesis Document Advisor

Brian Lucid
DMI Faculty

Toby Bottorf
External Reviewer
Abstract

The Articulation of Visual Experiences Through Algorithm explores the

concepts and possibilities computation presents as a creative medium for
design.

This thesis investigates the relationship between code and visual language.
From this relationship, a new vocabulary of new media will emerge,
driven by ideas that stretch their existence simultaneously through all
means – visual, aural, verbal, haptical, temporal and logical.

My methodology was based on discrete experiments native to the compu-

tational medium which investigated a range of topics including: musical
intervals; generative and interactive form systems; interactive op art;
database visualization; and the plastic components of time.

7
Aknowledgements

I traveled 4817 miles from São Paulo to Boston to improve myself as a

designer. This would have been impossible without the support of my
family, but I own absolutely everything to my wife Margarida, and her
love, support and patience.

I’d like to express my gratitude to Jan Kubasiewicz, not only my mentor,

but also a good friend.

To Joe, Gunta and Brian, for all the worthy hard work.

To my fellow classmates, Stephen, Keiko, Kate, Karolina and specially

Lynn (and Ronaldo) for all the support and friendship.

To my friends Colin Booth and BJ Jones, for trusting and receiving a

Brazilian weird student at the US.

To Dave and everyone at Beam interactive, for the opportunity of

practicing the thesis preach.

To Ben Fry, Mario Klingemann and all Processing community.

And finally to Google, for returning 264,000 thesis acknowledgments to

my amusement. There are surely a lot of grateful people out there!

9
Table of Contents

Definitions 17

Roots 27

Concepts 39

Praxis 53

Experience 67

Case Studies 71

Sound in Space 73
Form Systems 85
Riley Patterns 99
Logic Molecules 109
Patterns of Time 115

Findings 121

Bibliography 125

11
Preface

“Every work of art is the child of its age [...] Each period of culture
produces an art of its own which can never be repeated.”

— Wassily Kandinsky

13
Disclaimer

“To define is to limit.”

— Oscar Wilde, in “The Picture of Dorian Gray”

The limitation serves a purpose - better understanding.

This thesis document defines a series of concepts and ideas, many of

them vast, open and entangled. By any means those definitions intend
to limit concepts – they mean to raise understanding by disclosing
relevant issues.

Ideas are not – and shouldn’t be - limited to their verbal counterparts.

15
Definitions

Visual experience, defined.

Design is an act of assembly of form through composition.1

A composition that has a purpose: to communicate.

The idea of communication implies that a message will eventually be

received by someone; there is an audience, a receptor. Also, communi-
cation implies the existence of a medium, a dimension where composition
becomes reality.

Composition, audience and medium are linked together: each one influ-
ences and is influenced by the other, in a mutual, constant change.

The term “visual experience” is defined by the connection of these three

concepts; a composition acknowledged of its audience and aware to the
nature of its medium.

�����������

������

��������
a model for visual experience

���������������������

1
“Design is the method of putting form and content together. Design, just as art, has multiple definitions,
there is no single definition. Design can be art. Design can be aesthetics. Design is so simple, that’s why it is
so complicated.” – Paul Rand

17
Computer: a new medium

The personal computer has worked as a tool for design production and
artistic experimentation since its tender age. During the mid-nineties,
though, a shift in the role of computing occurred: the advent of the
internet, that allied with the proliferation of household computers gave
birth to a new mass communication medium.

Computational devices2 became a medium, with an important difference

from other known media – they allow direct feedback and participation
from the audience. The concepts of broadcaster and viewer got blurred,
since one can become the other after few keystrokes.

As a medium, computation affects the audience. Consequently, graphic

design is influenced by the change; the new medium opens a myriad of
possibilities for creation in time, space, shape and sound. The partici-
pation of the audience raises the concept of interactivity.

Computers have became no longer just a production tool, but also the
living space for the design creation.

New media creativity

Once a new medium has been established and the gates for new dimen-
sions in design have been opened, a question arises: how is the conception
of visual experiences changed? How can new media affect creativity?

The initial transformation in design was an adaptation; new media design

was a collection of items from different media bonded together. A print
design piece would receive navigation buttons, a small animation intro
and be published as a “new media” piece.

The adaptation evolved according to the features that mainstream

graphic software packages would offer; software companies like Adobe
and Macromedia ended up dictating the aesthetics of interactive com-
munications by offering “canned” solutions, like templates and plug-ins,

2
Although this document aims the exploration of personal computers, like desktop or portable systems, any
micro processed object is considered a computational medium: cell phones, palm tops, text messengers, DVRs
and others

18
and generating design elements according to their own proprietary algo-
rithms, like vector antialiasing and motion tween functions.

Up to this point in time, the creative process per se changed very little
– more dimensions were added, but the visual experience conception
remained the same. Under the navigation buttons and animated items,
conceptually print media pieces live online.

Birth

The computer is both a medium and a tool for design – but the creative
process seems to still be far from the machine. Why is it the case that the
principles of computation are not present in the creative framework, in
the conception of a visual experience?

They are present, but for a very small number of designers, still far from
the mainstream. There’s a gigantic gap between the number of designers
that use computers as a tool and the number of designers that use the
computer as a part of the concept in the design work. There are a small
number of design pieces that can only exist and be understood inside the
digital medium.

The language of design is in constant evolution, and now it faces the

opportunity to embrace the concepts and processes of its new hosting
medium, becoming native, being reborn.

What are the concepts of computation?

Computation is about change.

The personal computer is a highly evolved calculator, performing billions

of arithmetical and logical operations per second. The processor works by
getting certain inputs, performing calculations and outputting them; data
is constantly being chopped to small instructions, calculated and rebuilt.

Let’s use a keyboard input, for example; the physical keystroke is

converted to a binary signal; the signal is processed and converted to a
visual output, the graphical representation of the key – that is displayed

19
 ��������� ����������

������� �������
������ ������
����� ��������� �����
��������� ���������
������� �������
������ ������
a model for computation

on screen. The core of the computational process is about translating,

converting a certain input to a certain output.

The creative use of computation takes advantage of the nature of data pro-
cessing in favor of design. There are situations where traditional design
meets the limits of human labor; a computer can address a different scale
than human hands in artistic creation, dealing with the very large, the
very small, the very precise, the very exhausting, so on and so forth.

Some of the computation characteristics are:

• Transformation: as stated above, the main concept of compu-

tation is to transform data.

• Repetition: a processing instruction can repeat itself eternally, as

long as its physical part works. An element can repeat itself over time
or over space – the replication is lossless.

• Randomness: true randomness comes from nature, in events

like radioactive decay. Computer systems feature pseudo-random
numbers, a formula that produces no repetition, smooth numeric
distribution, and more important: the lack of predictability.

• Precision: computer processing elements can be extremely exact

and precise when necessary.

• Predictability: once a set of rules is created, the code behaves

accordingly. Just like the physical world is attached to the laws of
physics, a program can be attached to a set of unbreakable laws that
will be always obeyed.

20
 • Discretion: program elements can be discrete; events can be
triggered by any kind of input rather than linear time. This is the key
for interaction.

• Complexity: computation can achieve and display unbearable

amounts of visual or logical information.

• Simplicity: the opposite of complexity; the ability to complete a

task with a clear, austere and minimal output.

• Remembrance: the ability to store data, and recall it later, with no

logical capping of time or quantity – the only limit is physical.

The concepts of computation are implemented in a computer code as

formulas, or sets of instructions – known as algorithms.

The origins of the gap

There is a resistance in incorporating computer processing characteristics

in the concepts of design. The resistance has mainly two reasons;

The first reason is the difficulty in communication between humans and

computers; the computer language is different than common verbal
language. Some specific training is necessary so one can be fully able to
read and write computer code. At the same time, the amount of training
today is a small fraction from the amount of training twenty years
ago. Computer language is evolving at a similar speed of the computer
hardware evolution. Coding is more accessible now than ever, and there’s
a prognostic for future improvements. The challenge is to learn what to
tell the computer – and not simply how to tell it. Language syntax depends
on the available technology, but the idea to be coded is timeless.

The second reason is cultural and social; western education branches arts
and humanities from math and science. From high school to postgraduate
education, there’s a strong sense of separation between different areas
of knowledge. Computational design merges art, math, physics and even
psychology, when dealing with the study of interactive design reaction.
The union of different branches of knowledge is required – accompanied
by a mindset prepared to create blissfully in any necessary field of study.

21
Verbal, visual and computational

How different is the computational language from visual language?

The human skill of communication is granted by sensorial channels;

ideas, facts and feelings are understood and expressed whether in verbal,
visual or any other sense.

The compounding elements of a channel have similar counterparts on

another channel; correlations can be addressed between elements with
similar conceptual functions

The correlation of elements has a clear role in the understanding of

language; think about the word “red”. This word has strong visual in-
formation attached to it, representing the visual hue known as red. The
experience of red can be expressed verbally, through the word, or visually,
through the hue. It also can be learned both ways.

We can understand a concept by contextualizing it in a communication

channel; words can describe visual objects, phrases can describe visual
actions - and vice versa.

��������������� ���������������

������������������������������ ������������������������������

������ �����

���� �����

�������� �����������

correlations between verbal and visual domains

22
 ���

��������

������

��������

domain correlation examples

����������������������������
The computational language acts as an extra channel - it has similar
structural properties to other communication means, thus, correlations
of similar elements in the structure can be established.

The computer language, in essence a binary series of logical and arith-

metical operations, can be understood by contextualizing other channels
– especially, but not limited to, verbal and visual realms. We can commu-
nicate to a computer by typing verbal commands; the verbal will become
a digital process, returned and converted to any output – verbal, visual,
haptical, sonorous or any other available.

Language is a system4, where elements are composed to form meaning;

The structure of the computational language is similar to the structure of

visual, verbal or any other language; a structure of elements interacting
in a composition. Elements can be arranged to work together in a logical
environment just as in the visual or verbal form.

4
“A language is a system of pure values” – Ferdinand Saussure

23
 ��������������� ��������������� ����������������������

��������������������� ���������������������� ����������������������

�������� �������� ��������

������ ����� ���

���� ����� ������

�������� ����������� ����

correlations between verbal, visual and logical domains

�������
����������� �������������

���������� �������������

�������������
����������������
����������� ���������������
��������

domain correlation examples

���������������������������������������������

24
Module and System

Module is an independent part, a single element that exists by itself,

having built-in functionalities. System is the working interrelation of
modules, comprised as a whole.

Scale is a determining factor to define something as a module or a system;

an atom can be taxonomized as a module – a unique, independent part -
or a system, formed by the particles and sub particles that interact within
its apparent unity.

The building blocks that define a language are considered modules; each
module is unique, and represents a concept, an idea. Modules interrelate,
creating a system - the composition. Words interact in a sentence, shapes
interact in a canvas, and programming objects interact in a code.

������ ������

������ ����

���� ��������

�������� �������

������� ���� the structure of model/system,

with an example of application.

Articulation

The articulation of ideas in the visual, verbal or computational dimen-

sions is based on the composition of elements.

As initially stated, design is composition - acknowledged by its audience

and medium. The articulation of visual experiences can grow along the
creative potentials of the new computational medium;

It’s time to articulate visual experiences through algorithm.

25
Roots

Art, Science and Religion

The underlining concepts of computation and design have met many

times through history; accessible computation may be a new format, but
mathematical and scientific concepts aligned with art are not.

The branches of knowledge have been treated as one through time; the
knowledge of understanding nature, the knowledge of understanding
the spirit and the knowledge of expressing both. Science, religion and art
are linked together in the history of civilization. Pyramids, temples and
statues that remain still prove how engineering and art were combined in
service of belief.

a solid articulation of geometry.

Notable thinkers in history also show that the path of science and art can
be one; Greek philosophers like Plato and Aristotle, for example, inves-
tigated math and physics along with aesthetics and poetry – all inquiries
about the real world.

Jumping in time and space, one of the best examples is Leonardo da Vinci.
Da Vinci had a strong grasp of mathematics and engineering, showed
in his fabulous inventions, and strong artistic ability, proofed with his
invaluable paintings and sculptures. His creativity had no borders.

27
The renaissance era of da Vinci gives us an important demonstration on
the incorporation of mathematical and scientific concepts in the artistic
creation. Before this movement, medieval paintings were visually flat.
Space and volume were represented, but did not reflect reality. The artistic
composition, as a concept, was focused in the symbolical components;
the concept of composition was centered in the meaning.

Sassetta’s The Meeting of Saint Anthony

and Saint Paul:the vanished vanishing point

The renaissance marks the advent of perspective techniques. Perspective

drawing was initially a work of architects only, used in construction
projects. But the new aesthetic of space spread and many talented painters
embraced it when creating their work.

“Art as opposite of science? Masaccio, who is often hailed as

the first painter of modern times, was a member of the guild of
apothecaries and grocers”

— Karl Gerstner

In a non-digital way, Renaissance paintings are visual experiences ar-

ticulated through algorithm; the formulas of perspective were carefully
implemented when picturing space, scientific data was used to create
visual compositions and geometry became part of the creative process.

28
 Masaccio’s Trinity. Masaccio
employed complex mathematical
detailing when producing his work.

Advancing more in time, we arrive at the modern model of knowledge

categorization; each field of study is highly defined and specialized. As
civilization evolves, science grows in complexity – it’s intangible to grasp
all fields of knowledge. Art became a distant branch, and even the lower
stages of education impose a learning path choice.

“After a certain high level of technical skill is achieved, science

and art tend to coalesce in esthetics, plasticity, and form. The
greatest scientists are always artists as well.”

“All religions, arts and sciences are branches of the same tree”

— Albert Einstein

29
Physicists like Albert Einstein and Werner Heisenberg believed that
knowledge has similarities in different specializations – and there’s a
mutual benefit of connecting those similarities.

The Massachusetts Institute of Technology (MIT) plays an interesting role

in this relationship. Renowned for the development of groundbreaking
technologies in different fields of science, MIT maintains the CAVS
– Center for Advanced Visual Studies, founded by Gyorgy Kepes. Kepes
was a designer that did not believe in the separation of art and science;
his explorations on visual language and his books reflect this idea, and
for almost thirty years (1946-74) his presence at the MIT maintained this
ideal alive.

“An experimental effort to encourage such interthinking between

different disciplines in the visual arts and scientific and technical
fields is more than overdue. As the twentieth century has grown
older, such intercommunication has become seemingly more
improbable”

“The dynamic interdependence of the wisdom of the senses and

wisdom of reason is fundamental to a balanced growth”

“Yet, history shows us that art and science, two basic human
activities springing from a common love of nature are interde-
pendent. Each achieves stronger growth when cross-pollinated
by the other.”

— Gyorgy Kepes

Computation

The domains of visual and computational languages met right after the
advent of processing machines, during the fifties. The early explorations
involved tweaking the graphical output of huge computers to record film
frames and create animations, or by simply photographing an oscillo-
scope display that could be programmed. John Whitney, Sr. was one of
the most important founding fathers of computer graphics.

30
 “The union of color and tone is a very special gift of computer
technologies”

“Time has become visual”

“My computer program is like a piano. I could continue to use it

creatively all my life.”

— John Whitney, Sr.

These quotes regard Whitney’s explorations on computational abstract

animation and music1.

In 1962, A. Michael Noll at New Jersey’s Bell Labs created a computer

program that featured an algorithm whose function was to build visual
compositions; the reference for the algorithm rules was the composition
style of Mondrian.

Oscilloscope recording by Whitman

The work of Noll has extreme importance; besides Mondrian’s mimic, his
programs investigated composition from a probability stand point – visual
forms were derived from a computational concept. Lines would be drawn
(or not) according to the visual density of the composition. Other Noll

1 Two of his short movies can be accessed at

http://www.siggraph.org/artdesign/profile/whitney/nobio_intro.html

31
works dealt with geometric randomness or repetition; he clearly took
advantage of the newborn computation process in a creative way. The
machinery evolved a lot since then, but Noll’s ideas are still fresh – there
is no conceptual obsolescence.

At the same time, Douglas Engelbart was creating the foundations of

electronic communications and computational office work. Engelbart
developed the mouse, the hyperlink, video conference, file structure
navigation, word processing and more. Although he wasn’t pursuing
artistic objectives, his work supports the employment of creativity in the
computer medium, proposing to use the machine to augment the human
intellect.

Noll’s Mondrian experiment

The MIT made strong contributions in the graphic computing field. In

1963 the Sketchpad was created – the first CAD (Computer Aided Design)
interface. Sketchpad was extremely advanced for its time; the user would
interact with the application directly on screen, with a pen, through a
graphic interface. All the interaction concepts of this system are the
basically the same we use today.

In the early seventies, Xerox Corporation founded the Palo Alto Research
Center, also known as PARC. It was inside the PARC that most of the
“modern” concepts used in computation were invented, like the icon,
object oriented programming, laser printer, WYSIWYG (what you see

32
 The SketchPad

is what you get), computer notebooks, paint and animation applications

(before that, everything had to be coded or use dedicated hardware), midi
composition and many other ideas that shaped the terms “multimedia”
and “graphical interface”. The PARC computers operating system was
called SmallTalk and it featured something that is still lacking in today’s
operating systems: seamless integration between the graphic interface
and the code behind it. The user was able to customize, change and create
new functions whenever it was necessary, since there was no barrier to
explore the operating source code. PARC successfully conducted experi-
ments where teenagers were taught to code in SmallTalk language, and
soon they would use their creativity to generate graphic applications.
SmallTalk can be considered the golden age of computational design,
when the visual and the logical sides were always connected and evident.

The contemporary contribution from the MIT comes from a new gen-
eration of scientists/artists/designers developing projects regarding the

PARC SmallTalk - 1976

33
interaction of humans and technology. This happened at the “Visual
Language Workshop”, in 1973, and later, in 1982 when the Media Lab was
founded by Nicholas Negroponte. The specific topic of visual language
mixed with computation was more intensely researched years later by
Muriel Cooper, a pioneer in digital typography who worked to develop the
language and structure of electronic design, and to expand the graphic
design culture to four dimensions. John Maeda, considered Cooper’s
successor, founded the “Aesthetics + Computation”2 group, where many
researchers were involved in visual issues through programming, and how
the visual language could be expressed and interacted with an audience
through computation.

The Aesthetics and Computation group started its operations at the

same period that the internet blossomed and desktop computers became
present in common households. Designers started to create for the
computational medium; the graphic software packages would keep the
programming language underneath a friendly interface, “protecting”
designers from the necessary internet coding. The edge of programming
for artists was consolidated with Macromedia Flash and Macromedia
Director, commercial software packages where advanced users could
script their own visual behaviors code.

The term “Aesthetics + Computation” wasn’t just university lab concept

anymore – it was present in the society. One of the projects from Maeda
dealt with accessible computer coding focused in visual elements, called
“Design by Numbers” – a coding interface whose strength was in its
minimalism, where the communication between designer and machine
could be clear. Maeda’s project appears to have strong influence from
the previous “Turtle” project from MIT, a software application where an
element on screen called turtle could be moved through code in order to
build images. The turtle project later was know as Logo, and was largely
used in schools during late seventies/early eighties. Logo helped to teach
the basic concepts of computing – the same idea that Design by Numbers
explore. John Maeda’s work also helped to spread the idea that design
could be achieved through computational means, that creativity could be
expressed through code with his interesting explorations of algorithmic
visuality.

2 http://acg.media.mit.edu/

34
 Maeda’s Design by
Numbers interface

Logo interface

The second project from MIT towards a visual creation code was
created by Ben Fry and Casey Reas, and is called “Processing”. It adds
a great number of features and extend “Design by Numbers”, providing
more creative freedom and keeping the accessibility for non-computer
scientists. Processing has today around 1000 users – a small number
compared to the whole interactive design community, but a big number
of participants in a non-commercial, non-advertised university project,
that is still entering its beta stage of development. The most important
fact: the massive majority of the users are interactive designers, who were
initiated to scripting languages when developing for the internet, and
sought ways for doing more. Processing has no timeline, no visual thrills,
and its interface is a simple text editor. But the concept of the application
is clearly targeted to the artistic creation, hence its success among the
design community.

35
 Processing Beta interface

36
Concepts

Creative Framework

The idea of a creative framework can be explained through examples; let’s

use painting and photography. Two forms of expression that share similar
characteristics – they’re visual, compositive, bi-dimensional, have color
hues or tones that form shapes and they communicate through visual
information.

Although similar, they’re also very different. It’s easy to discern one from
another; they have different aesthetic values, and generate completely
different reactions.

Photography strongly reflects reality, augmenting or diminishing an

aspect of it by light, texture and positioning; photography is subtractive
– the artist takes the physical world and frames a theme out of it, a part
out of the whole. The kinds of camera, lenses, time of the day or season of
year are manipulable elements that the photographer can use to achieve
his intended visual result.

Painting has its strength on expressing the author’s internal world; even
a photo-realist painting translates the artist’s view of light, color, form.
A painting is built from a clear canvas; it is additive, colors and shapes
are added to build a visual piece. Pigments, solvents, brushes and stroke
techniques are elements that the painter can manipulate on creating his
work.

Both examples - painting and photography - are visual experiences; com-

positions that communicate in the visual realm. But they’re built through
different creative concepts, using different creative tools.

Creative concepts are the mechanics of expression in a medium – the ways

that an expression can become reality. Light is a concept in photography;
space is also a concept. Both determine how the visual experience of a
shot will be. In the same way, shapes, colors and dimensions are concepts

39
in painting; each medium has its own set of procedures. The instruments
that allow those concepts to materialize - lenses, brushes, pigments, film
– are creative tools.

Computational Concepts, revisited

As stated initially, computation is about change - and its mechanics feature

concepts like transformation, repetition, randomness and others.

Pure computation, just like pure mathematics, is an abstraction; it does

not necessarily have a component in the visual domain in order to exist.
If a piece of code is written on screen, the written letters are just a visual
representation. The computational language can be reduced to a series
of electrical signals traveling trough logic circuits. The elemental state of
computation is a traveling electron.

It might appear difficult to correlate such abstract medium to the creation

of visual experiences, but we can compare it to cinema; the nucleus of the
moving picture is time –the articulation of visual experiences through
time. What is the visual component of pure time? Can time be touched,
seen? Time is an abstraction that applied to the photographic medium
produces montage – composition in time, a movie.

We are able to understand pure abstraction by contextualizing it in a

tangible medium. Film, for example, contextualizes time through image
montage - flashbacks, slow motions, time lapses - the film strip is able
to demonstrate properties of time visually. The pure abstraction of
computation can be contextualized - visually, through pixels, or verbally,
through code. The intangible “traveling electron” can, finally, generate a
visual experience.

40
Modules

Scale determines if a concept should be understood as a module or as a

system – the category depends on the point of view. In a standard user
level point of view, the general computational module is called data.

Data is a computational fact, the declared digital information. The

category of data is called data type. There are two kinds of data types:
primitive and composite.

Primitive data is single information, attached to a value. Numbers are

primitive data. There several categories of numbers, according to the
amount of bytes they take into the computer memory. Logical states (true
or false) and single characters (“a”, “b”, “c”) are also primitive data. The
value of a declared data can be fixed, or vary according to a function; data
with mutating values is simply called variable.

��� 1
���� 1.0

���� 1e20
������ 1.77e-3

������
“a”

����
true
false
��������� �������
����

��������� ������ “hello”

�����
{1, 2, 3, 4}

�����
class example
{
var example1;
var example2;
}

41
Composite data is the name given to primitive data combined; a word,
for example, is a combination of characters, and is called string. Arrays
are another kind of composite data: they’re groups. They can be groups
of other groups (nested arrays) number, characters, objects or any other
datatype.

Classes and Objects are composite data. They’re an important concept

to be grasped, since almost all modern computation is based on Object
Oriented Programming. Object is a philosophical term: it defines
something, a particular item that has properties and relations. A software
object can be compared to a physical world object, or a visual object.

> this.teaPot

a beautiful physical object a reference for a logical object

Every object belongs to a class: the class is the template, the category that
defines the possible properties that an object can have.

class Teapot {
float size;
float color;
String lid
String spout
}

things the make a real teapot class “Teapot”: things the make
be a teapot: a teapot object be a teapot.
1.ability to hold liquids The initial defined properties
2. handle are size, color, the kind of lid
3. lid and the kind of spout.
4. spout

42
The main feature of programming through objects is the modularity:
objects can be multiplied, reused, changed, and are independent beings
inside a system.

Another important computational concept is the function. A function

describes an action to be taken, the relation between elements to be
realized; it drives the change on any kind of computational module.

Modules in action: system

Computation is about change; through functions or other methods,

computational elements can be put to work, perform a concept – act as
a system.

Let’s review some of the systematical concepts of computation:

• Repetition

A system is never truly static – except when data is stored in a physical

media, its existence lies within electrons. A static module is an illusion: it
is a module that is repeating itself over time, reaffirming its existence on
each circuit cycle.

The invisible repetition can be visual, when a graphic element stands

in time - or repeats itself over space. The simultaneous repeated copy is
called instance.

In the following example called repetition.pde, coded using Processing,

a rectangle object is repeated over space - and infinitely, over time. The
code also performs two iteration loops to proper position the instances
along a grid.

void setup() {
size(100,100);
fill(0);
}
void draw() {
background(255);
for(int i=0; i<3; i++){
for(int j=0; j<2; j++){
rect(10+i*30, 25+j*30, 20, 20);
}
}
}

43
• Transformation

When a property of a module is being changed over space, or time, or

any kind of input, a transformation is occurring; almost everything in
computation is an example of transformation.

The following simple code called transformation.pde increases the value

of the x coordinate of a square over time. When the edge of the screen is
reached, the value is reset, and the transformation starts again.

int x = 0; // coordinate in x axis

void setup() {
size(100,100);
fill(0);
}
void draw() {
background(255);
x = x+1;
if (x>=width) { x = 0; }
rect(x, 40, 20, 20);
}

• Randomness

As initially written, true randomness comes from uncertain events in

nature. A known source of uncertainty is the radioactive decay, or the
movement of gas atoms in vacuum. Randomness is a controversial topic
in physics and mathematics, but no technical background is required to
use the idea of unpredictability in while designing. Random algorithms
produce a number inside a certain range; the range can be a space
coordinate, size, or a moment in time. Any property can be tweaked in
random ways.

The following example (randomness.pde) generates a random coordinate

for the square each frame:

float x; // coordinate in x axis

float y; // coordinate in x axis
void setup() {
size(100,100);
fill(0); }
void draw() {
background(255);
x = random(width);
y = random(height);
rect(x, y, 20, 20); }

44
• Precision

The generation of images through algorithm sometimes requires a great

deal of mathematical processing; certain values go through hundreds,
thousands of calculations – and the visual output depends on how exact
those calculations are perfomed.

The case study of the harmonograph (page 78), for example, shows
musical intervals converted to images; the purity of a interval determines
if the image will be symmetrical and stable or not. The visuality of the
whole piece changes with any small deviation of the interval value, even a
0.0000001 unit. The concept of precision thus is valuable when obtaining
certain visual pieces.

harmonograph: any change in the

interval value creates a different visual
representation

• Predictability

Predictability comes from nature; for example, it’s absolutely predictable

that all physical objects on Earth will obey the laws of gravity. Physical,
chemical and biological processes occur with confidence that they were
supposed to happen. In a computation system this concept can be
recreated; a function can set behaviors to any module, and this behavior
will happen, always, while the system is active. Creatively, it allows the
designer to create his own rules: like that all objects should vibrate all
the time. Or everything should become red when touched and so forth.
Computer bugs are the outlaws of predictability.

45
• Discretion

Being modular and being discrete are almost synonyms; events in com-
putation don’t need to be linear; events can happen in modules – they can
be handled through any kind of input. A computer code can have a set of
functions, and these functions be called to act any time. A user interacting
with a mouse is an example; when the mouse clicks an item, an event is
called.

The mouse was the event that determined the moment in time where
the action took place; that action could happen again and again – or not.
There’s no need for linear time in logical modules.

• Complexity

The concept of complexity is to use computation to achieve a degree of

element profusion and element interaction that is intangible to human
hands; based on the principles of repetition and transformation, com-
plexity can become more an aesthetical statement, searching beauty in
the intricate.

Peter de Jong
algorithm shown with
2,000,000 iterations.

• Simplicity

Simplicity is the holy grail of computation, since the complexity is a

natural tendency; bulk processing powers opens extensive creative
options. A visual experience can be successful either from complexity or
simplicity.

46
 Malevich’s Black Square and Red Square :
simple and powerful.

• Memory

There are two kinds of memory in a computer system; the volatile, logical
memory – where the necessary data for realizing a code is stored, and the
physical storage.

The logical memory has a temporary nature; to perform operations, the

computer stores data of any datatype in its functional memory banks.
A computer can retrieve with precision any item from its electronic
memory.

The physical memory is permanent1; the role of time is very clear on the
concepts of computation, and the physical memory is the place where
time stands still. Like a printed book, binary data is printed onto an
optical or magnetic surface, where it can be physically stored.

The creative value of storage is in its power: a database can hold an un-
imaginable amount of information, in any kind of format. The internet
itself is a gigantic database, that can be retrievable through search engines.
The Human Genome Project and the CERN nuclear collision database are
examples of huge unique data storage projects, with billions of bytes that
can be retrieved, sorted and shaped in any way.

1
Subject to physical decay through time; under controlled storage conditions, the decay is minimal.

47
 Ben Fry’s Haplotype Lines: a creative visualization of the massive genomic database.

The concepts of computation can be visualized, or verbalized, in countless

forms; the key to achieve visual experiences through them is to employ
artistic creativity in all imaginable – or unimaginable - ways.

The issue of generative design

Generative design is a term vastly employed to describe visual pieces

generated through computational processes; but its true concept is not
clear. Between its diverse definitions, it can be used as “design generated
by a computer” or “design generated through a computer” – very different
ideas.

Can a computer design?

Algorithms can generate visual compositions from ether; visual elements

can be built and tweaked in any amount, following all the concepts of
computation. The illusion that a computer is designing comes from the
capacity of variation and randomness on the construction of “new” com-
positions.

Sadly, random and abstract are not synonyms. Abstraction is a thought

process. Computers don’t think (at least yet).

48
The computer intelligence is a widely discussed topic; one of the most
famous writings on it comes from Alan Turing, a mathematician that
published in 1950 a test where machines would be considered intelligent
if they could engage a non discernable conversation with a human being.
The validity of the Turing test is questioned today, but no machine could
achieve the “victory” so far. Despite the advancements on artificial intel-
ligence, it remains artificial - a mimic, or a source of algorithmic varia-
tions.

The conception of visual experiences is lead by original ideas; algorithms

can be programmed to follow the principles of visual language – pro-
portion, distribution, color harmony; they also can be programmed to
follow style guidelines – use certain color harmonies, certain line styles,
and achieve an overall aesthetic sense. But the leading idea remains in the
hands of the designer.

�������������
������������������� ����� �������

�������������������� ��������������������

�������

��������������� �������������������������

��������������

���������������� �������������������������

The term generative design shouldn’t be used to define design created “by
the computer” – it should be used to describe “through the computer”
instead. In a broad sense, any design that is visualized through a computer
can be considered generative, but the usage of the term becomes more
appropriate when describing automated processes - the computer is
programmed to build automatically certain visuals according to a set of
variables and pre defined laws with no need of human interference.

49
 “…the language of the computer is the language of technology,
not the language of design. Without a knowledge of design, the
computer (like the pencil) is more than useless, for it is capable
of producing enough superfluous material to create the illusion,
that one is inventing when in fact, one is merely producing varia-
tions of a theme, often of nothingness.”

— Paul Rand, Design Form and Chaos, 1993

50
Praxis

The concepts of computational design need to be applied somehow; In

order to transform abstract concepts into real projects, tools need to be
used.

The computer is obviously the most necessary tool for the task. It needs
to feature a screen – a device capable of displaying an array of pixels,
light elements that are lit or not to build shapes and colors. It also needs
to feature input devices, like keyboard, mouse or joystick. Finally, it
needs the software, a logical layer between the computer circuit and
the designer. The choice of software is a controversial issue: different
packages offer completely different experiences for the designer and
for the audience. Also, every software title imprints different aesthetical
details to the pieces generated by them.

A software title that helps the creation of visual experiences will act as an
interpreter, a translator between the designer and the machine. Different
interpreters will build the translated content in different ways; the overall
meaning might be the same, but the lexical construction will differ. Also,
software packages use libraries of functions – pre written formulas that
help the development of a visual piece.

When the functionality of pre-written libraries grow, an effect called

pasteurization might happen: little individual differences in the code
between one author to another are suppressed; the code, thus the visual
(since one generates the other) gain a very similar aesthetic aspect. They
look the same. Commercial packages often break the barrier of canned
aesthetic subtleties and offer completely industrialized, homogenous
visual solutions - templates. Less, easier work – but evident less visual
value in design.

53
Forms of Code

Canned or artisan, the software tools provide a clean slate for creativity.
There are many options to design through algorithm and these options
have mainly three categories: compiled, scripted (aka interpreted) or
markup languages.

Their definition is a little fuzzy; the main idea is that a code written in
a compiled language is converted to machine language – “compiled” -
before it is distributed and executed. The final product does not contain
the original written code, but a translated machine code. For example,
C and Pascal are purely compiled languages. C++ and Java are semi-
compiled, they stand in an intermediate structure between compiling and
scripting modes.

Scripting languages are interpreted at run time; the code is converted

to machine instructions at the same time the program is running. The
distributed/executed code contains the original language written by the
programmer, and it needs the support of a compiled structure to receive.
The performance of scripting languages is worse than purely compiled
ones, but its flexibility is greater; Postscript, JavaScript and Flash’s
ActionScript are popular examples of scripting languages.

Markup language is the name for content that is tagged to be correctly

displayed and interpreted. HTML and XML are markup languages – texts,
pictures or any data with the correct tags for the future interpretation.

Forms of Image

Another important distinction to be noted in graphic software is the defi-

nition of bitmap and vector images.

Vector graphics are shapes defined in the computer memory by the

formulas of its form, the formulas of how lines and curves are posi-
tioned and connected. They became very popular on the web thanks to
Macromedia Flash, and very popular in print design thanks to Adobe
Postscript. Vectors can be written as a map of coordinates in two or
three dimensions, creating 3d models. The vector formulas need to be

54
converted to pixels before being shown on the screen; depending on
the software that is interpreting them, different rendering results are
noted, different visual styles are achieved. Pixels are square elements,
and they’re required to represent points in curved shapes; The rendering
engine needs to decide which pixels should fit in the curved lines, and also
decide how the jagged edges can be smoothed (also know as antialiasing)
to complement the illusion of curviness

a vector image, and a blown-upof its pixel representation.

Raster (or bitmapped) graphics are visual files where an array of pixel color
values is stored; all the pixel values together form an image. Photographs
are examples of bitmapped images; bitmaps can be compressed through
algorithms that perform pixel statistic operations, discarding information
that it judges not relevant to the image and making them smaller. Bitmaps
are already made out of pixel information, so unlike vectors they don’t
need to be converted to be displayed on screen.

Software

The most common software tools available today for the different code
form categories are:

• Adobe Photoshop/Illustrator/After Effects

the Adobe software titles are used as graphic production tools, not
coding environments; but the reality is that algorithms can be written

55
 and implemented on then. After Effects, the broadcast motion graphics
standard has support to the common JavaScript. Illustrator can
import postscript files that can be generated through code; Photoshop
also can have plug-ins developed, automate functions through scripts
or deal with postscript files. All these coding doors can be taken as
advantage in the creative framework.

• Macromedia Director

One of the oldest multimedia authoring platforms; Director is first of

all, an aggregator: it can unite almost any kind of media asset inside a
project. Director has a powerful, but very extravagant code language
called Lingo - extravagant because its syntax is very different from
the coding standards used by other popular multimedia applications.
Director has full support for real time, hardware accelerated 3D
graphics and is largely used in complex networked projects, where
flash is not powerful enough.

• Macromedia Flash

Flash is one of the most successful pieces of software in history; the

initial plug-in was designed to visualize vector graphics in a web
browser, and animate them through tiny file sizes. The original
l996 software called “CelAnimator” evolved, having more and more
features added every year. Flash is a program that can be used in in-
numerous ways – as a linear animator, as an illustrator or as a creator
of applications and web sites. Flash has a scripting language called
ActionScript, in part responsible by the actual movement of designers
working through code. It is necessary to learn ActionScript in order
to build interactive movies; the basic scripting of a site navigation
is an easy task for non programmers – which mean a smooth tran-
sition from traditional design to the possibilities of design through
code. ActionScript can be very simple, but it is also capable of doing
complex tasks as the user learns more coding techniques. There are
limits in flash though; the package still does not offer control of the
actual pixels being drawn on screen; the user can only code through
vectors, and the closed flash engine process the conversion to screen
pixels.

56
 • Java

Java was created in 1995; its initial concept was to be an extremely

portable language that could be executed in any computational device
– from desktop computers to kitchen fridges. The Java-enabled
consumer electronics were never fully employed, with the exception
of cell phones, but the technology turned out to be very useful for
network applications – including the web. Java is a programming
language modeled after the Object Oriented concept, and is con-
sidered a “semi compiled” language, designed to be easily ported to
different computer systems.

• Processing

Processing a programming language created aiming artistic and

visual applications developed through code. Programs generated
through Processing are usually called sketches, and can be run in any
computer platform, or over the internet. Processing is built over Java,
having added its own graphic library and functions. Java language
has a steep learning curve, and requires lots of technical background.
Processing does not have a graphical input interface; the sketch is
designed in a textual way, but the logic within it is very visual, making
sense for visual oriented users, as artists and designers. Processing
grants pixel-level control; the screen array can be manipulated
directly. Also, it has great extensibility through additional libraries
– like sound, midi, HTML or physical interfaces. Almost anything can
be integrated into a artistic sketch project.

Every software package demands time from its user to fully learn all the
concepts, functions and possibilities. The technology of the future is almost
unpredictable, making hard to decide which platform will have more ac-
ceptance and more capabilities in few years. Macromedia Director, for
example, was really popular for the CD-ROM production in the nineties;
its usage has been decreasing, being now restricted to some specific 3D
projects – and it’s likely to become less popular in few years. Other factor
to consider when learning a language is the “commercial versus free”
software factor; commercial packages usually have, at least in theory,

57
more technical support, because they’re products being sold to customers;
the designer is paying the developers. In fact, free software communities
end up having a good network of users who support each other, even in
a better way that paid application service. The real advantage on com-
mercial titles is the marketing that convinces users to adopt means that
will allow projects to be seen. Recently, Microsoft decided to cut the free
Java as a pre-installed plug-in for the Explorer browser; meanwhile, the
Macromedia Flash player still comes pre-installed, and is present in 95%
of the user browsers.

Using Java/Processing as a creative tool also features the advantage of

controlling the visual rendering process to a low level depth. As said
before, pre-programmed functions end up having an aesthetic influence
to the visual product; programs like Processing allow the designer
to customize the aesthetic imprint by writing their own set of visual
rendering functions.

still from Bézier Ballet, by Mario Klingemann

(http://www.quasimondo.com).
Unhappy with the Processing default
antialiasing algorithm, Klingerman wrote his
own, capable of displaying the smoothness
and delicacy of thin bézier lines.

58
Mathematical Tools

Each software package has its own features and its own coding syntax to
be learned. But one thing is common among them all: Math.

It is necessary to know some math concepts in order to create visual

experiences through algorithm; and this is one of the reasons that people
are driven away from this kind of creative process. Art education has been
far from Math, and mathematical ideas are seen as impossible problems
by most of designers.

Happily, the truth is that the necessary math for visual creations is very
tangible, and all it requires is the high school math remembrance. 95%
of the creative visual work produced through algorithm today employs
high school level math only, including algebra, trigonometry, physics and
geometry.

The key is not to know complex math; the key is to use creatively the math
you already know. Math is a tool, and the complexity of the tool never
meant the quality of the creation – neither in arts or sciences.

The Leica camera used by

Cartier Bresson had only basic
photographic functions. It is
a quality simple instrument,
the tool of Bresson’s beautiful
photographic expression.

59
Remembering high school

What is the basic math a designer is supposed to know?

A good general review at the high school books is recommended;

sometimes, easy formulas or concepts that may be forgotten can help
to transform an idea into a visual. There are some topics that deserve
special attention; a good foundation on trigonometry, for example, will
solve a great number of issues related to circular motion and visual oscil-
lation. The Pythagorean Theorem, that solves the right triangle lengths
(c2 = a2+b2) , will also solve questions regarding positioning, distance and
linear motion along the space of a visual piece. Basic operators (addition,
subtraction, division, multiplication) can also have creative potentials
- since a block of code will loop indefinitely, and a number can go through
an operation and change its value on each cycle. The modulo (the operator
that returns the remainder of the division between two numbers) is one
of the forgotten high school arithmetic topics that can have a great use on
computation – modulo values are cyclic, they repeat themselves when the
operand is constantly incremented in a loop. Colors, forms and position
can be cycled along the modulo result. The author of this document knows
very little math beyond the topics related above – and even so, produced a
series of intriguing mathematical visuals; the best advice to overcome the
Math-phobia is to roll up the sleeves and work on visual projects – learn
the math that will solve each specific visual inquiry, one at a time.

Following, a code example built in Processing that shows the application

of modulo and trigonometry; a box rotates in circular motion – achieved
by calculating the x and y coordinates according to trigonometric rules;
its color switches to black and white according to the modulo value of the
loop cycle. The sketch can be found on the attached DVD under the name
Trigonometry.pde.

“Art and Mathematics are complementary different; that is to

say: they differ each other just as much as they complement each
other.”

— Karl Gerstner

60
 create a loop counter variable: float counter;
create a circle radius variable: int radius;
create the x and y coordinates variables: float x, y;
configure the piece: void setup() {
set the velocity of the animation: framerate(20);
set the dimensions of the sketch: size(200,200);
set the initial value of the counter: counter = 0;
set the value of the circle radius: radius = 50;
end of the configuration block: }
loop starting point: void draw() {
set background color: background(122);
modulo operation:
if the value is even, fill the square with black: if(counter%2==0) { fill(0); }
if odd, fill it white: else { fill(255); }
x axis value according to trigonometry circle: x = sin(counter/10)*radius;
y axis value according to trigonometry circle: y = cos(counter/10)*radius;
set the rectangle coordinates and size; rect (90+ x, 90+ y, 20, 20);
increase the counter value: counter = counter+1;
end of the loop block: }

The Code Language

There’s not a universal language for computer coding; each coding

platform has its own syntax. Different languages can share similarities,
but they have different ways to handle the code structure, making them
unique - and requiring unique training.

Someone who’s aspiring to write code should choose which language fits
best to his needs; for commercial internet creation, Flash is surely the
best choice today. For artistic experimentation, Java and C++ are better
options, since they offer less limitations and more performance than
Flash. The Processing language has a lot of advantages for the artistic
field, including strong community support and a balanced learning curve.
There are many other options – Python, C#, Objective C… all computer
languages can be a source for creative work.

There’s also no single method of learning the code syntax; each author
proposes a different path through the functions and commands of each
platform. A recommendation for the learning process of a visual-oriented
user can be something like:

61
 • learn to compose a basic code structure – setup and loop blocks.

• create and configure the visual space for a piece (the stage)

• learn the methods of drawing basic shapes;

• learn the methods of tweaking the attributes – color, size, rotation

– of those shapes;

• learn how to move things around.

• learn how to make the user interact with elements.

This sequence would start out of graphic functions, instead, for example,
the common foundation on Object Oriented Programming – that involves
more abstract and less visual thinking.

The Processing platform has a lot of support for learning from the artistic
point of view; the language reference includes simple graphic examples
on the syntax of each topic at http://www.processing.org/learning and
http://www.processing.org/reference.

62
The Praxis of Coding

The process of writing code is similar to the process of writing a narrative

text.

A narration tells a story; there are characters, each character has a role
and things happen as the story develops, coming to an end. The same
narrative story can be written in different styles, different languages;
it can be written in a very erudite form, or it can have a teen literature
lexicon.

A computer code with a determined objective also can have different

styles, and different structures. The style and structure of coding can
be very personal; the way variables and objects are named, and how an
algorithm is written or a problem is solved. But there are principles that
should be followed.

The basic principles of coding are simplicity, clarity, generality and au-
tomation.

By definition, computation is a problem solving task. And there is usually

more than one way to solve a problem; the principle of simplicity is
that the simpler way to solve a problem is the best – less computational
resources are allocated, leaving more room for other operations.

Clarity is an issue for both man and machine; code should be clear for
other people rather than the author to read it, and for the machine to un-
derstand it. The clarity is achieved through good naming practices, using
parentheses to clear ambiguities (sometimes its use is not mandatory,
but helps the human reading), commenting the chunks of code, and
describing any relevant information in the program body. Another item
for clarity is the avoidance of contradictions; for example, the usage of
lower and uppercase. A variable named “Test” and a variable named
“test” are completely different in a case sensitive environment. If a piece
of code is using a variable in lowercase, all references to it should also be
in lowercase.

Generality is a principle that follows the spirit of object oriented pro-

gramming. A program can have very specific functions for each task;

63
sometimes, these functions are similar, and could have been written as
one and reused. General functions allocate less memory in a computer
system and allows better performance of the code.

Automation is a principle where the computer should handle repetitive

tasks, not the designer. It is an idea related to the principle of generality;
Instead of writing a commands or functions that perform a similar task
hundreds of times, it is better to write one function that operate the same
way with different parameters, saving time on the development and
making the debugging process easier.

A good practice for someone who is not comfortable with the task of
coding (or for someone with a very complex task in hands) is to first write
the program statements in plain English; list all the actions, variables
and describe all possible functions. With a cohesive and well structured
English text in hands, just translate it to the programming language
syntax – and the code is done.

64
Experience

“Graphic design which evokes the symmetria of Vitruvius, the

dynamic symmetry of Hambidge, the asymmetry of Mondrian;
which is a good gestalt, generated by intuition or by computer,
by invention or by a system of coordinates is not good design if it
does not communicate”

— Paul Rand

A Visual Experience involves the audience; designing a visual experience

does not contain only the “what to do” issue and the “how to do” issue; it
also contains the “whom to do”, the concern on how a visual piece will be
experienced by the audience.

The experience is an individual process; each person responds to a com-

munication stimulus in a particular form. The creation of visual expe-
riences deal with probabilities; defining the most probable response to
each presented stimulus, and planning how the visual piece will develop
according to the most probable situations.

There’s no real passive audience; if a visual piece is not interactive, there’s

always a choice for someone to see it or not; to ignore it or enjoy it. The
act of flip a page or not, close a browser window or not is a basic level of
interactivity.

Interaction is any reaction to the design stimulus.

A positive reaction is the successful communication. The designer should

compose a piece in visual terms but also in interactive terms; how a piece
will be originally seen? Is there a waiting time to see it? How much infor-
mation is given at each step, each section, each second?

Dosage: the concentration of information on every design node, part,

section, screen – considering information every visual, aural or any media
output. The design of interaction deals with the dosage of how and when
things will be experienced along a piece.

67
Regarding the visual terms, the shape of information should raise identity.
Identity means that the each person in the audience will recognize aes-
thetical ideas in common between him and the presented experience. The
recognition can be simple as a “I like it” response – and it does not mean
that the user identifies the output as something already seen before,
but as something in his personal taste, a match to his preferences. The
aesthetic values should be shared between both ends of communication,
the designer and the audience.

The development of visual experiences through algorithm opens doors

of limitless customization, interactivity and the generation of customized
information channels: each piece can be designed and react according
to the viewer. Despite of the unpredictability of a human response, the
audience should be acknowledged during the creation process - the
designer should not lock himself in an ivory tower, design is about public
experience – design is about communication.

68
69
70
Case Studies

The following projects were developed during the MFA program at

MassArt; they relate, identify and explore issues on the task of articu-
lating visual experiences, and they are the product of a designer – whose
education was centered in humanities – dealing with computation as a
tool for the visual domain.

71
72
Sound in Space

“Music is geometry in time”

— Arthur Honegger (1892-1955, Swiss composer)

What’s the relation between the aural and visual domains?

Is there an image for sound?

Sound is powerful enough to survive without our eyes. Music is a legitimate

expressway from and to our souls; music strikes the heart with no need
of visual alliances. At the same time, once we add a visual component to
sound, a relationship is formed - visual and aural domains stick together,
building an undeniable bond.

The bond can be formed in two ways; the first is by converting the
character of the musical signal into a visual representation; Apple Itunes
does this very well. Its visualizations uses the pitch and volume to feed
graphical algorithms. The second way evokes the meaning of music along
with the meanings of the visual domain; this idea is seen (and heard) in
cinema. A dramatic scene has a dramatic soundtrack; “dramatic” is an
adjective for the musical meaning; the pitches and volumes alone are not
dramatic – but the musical movement altogether can be.

The relationship of visual and aural has been explored throughout history
in many different ways; in eastern civilization, one of the oldest scientific
approaches to it comes from Isaac Newton. In 1672, he studied the color
spectrum and related 7 colors to 7 musical notes. Later, Louis Bertrand
Castel built the first “visual keyboard” – the Clavecin Oculaire – that
displayed a different color to each played pitch. The idea of a keyboard
that would display colors for each note reappeared in different places and
moments in time; there was, though, never a firm proof that a certain

73
color is a certain pitch. The relationship always had an arbitrary element
in its formation. Different color/pitch tables were developed, and many
authors agree on the color distributions – but there was never a scientific
standard. Intuition and feeling, personal factors, remain present.

Besides color, sound always had a strong relation to math and geometry.
The notable study of this relationship is harmony. Harmony is the study
of the simultaneity of sounds; Pythagoras, more than 2,500 years ago
started to study why some sounds had a pleasant character when played
together. Pythagoras studies had a strong mathematical sense, and he
established the concept of intervals.

Each musical note is a vibration; each vibration has a frequency. Our

perception of pitches, or musical notes, has a cyclic nature – every time
a frequency doubles, we perceive it as the same pitch, on a higher tone.
It’s called an octave – the space between two frequencies with the same
note. Numerically, the octave interval is notated as 2:1 proportion– the
frequency of the interval note is 2 times bigger than the whole funda-
mental note. A fundamental note can be divided in any other proportion,
and some of them are used as standards in music, like 3:2 (1.5 times bigger
frequency than the fundamental - fifth interval), 4:3 (fourth interval), 5:4
(major third), 6:5 (minor third), so on and so forth.

���������

����������

��������

����������

���������������

���������������

the proportional lenght between some musical intervals.

74
A mathematician called Jules Lissajous developed an experiment in 1857
that projected a vibration movement through a light dot in a screen. He
managed to reflect the light beam through 2 mirrors, each one vibrating
in an axis, vertical and horizontal. The resulting projected shapes became
known as Lissajous Figures

original Lissajous experiment

Different vibrations could be applied to the mirrors, producing different

shapes; musical notes are vibrations, so different pitches could be
reproduced and visualized - and when different pitches were projected
together, musical intervals could be seen.

The Lissajous figure might be the most scientific approach to a visual

representation of sound; it is based on the physical properties of musical
notes that cannot be easily contested like the previous color/sound ex-
periments.

75
The Chordinator

Chordinator interface screen

The Chordinator re-creates and expands Lissajous experiment as an

interactive computer application. Lissajous projection was limited to one
single interval; two notes, because his physical projection system could
only be bi-dimensional. The starting point of the Chordinator is to take
advantage that a computer system can create the illusion of depth easily.
There are 3 spatial dimensions that the human perception can discern:
height, width and depth. Musical chords are 3 (or more) notes played
together; two intervals in the same period of time, in a chord structure (
root, II degree, III degree, the button labels) The idea is to assign a spatial
dimension to a sound frequency, and explore the plasticity of sound as a
sculpture.

76
The sketch allows the independent control of the intensity of each chord
component, as well as its phase. Phase can be understood as the coordi-
nation point between sound waves

phase example: three sound

waves with same amplitude
and wavelenght with different
���� alignments in time.

To help the sense of 3 dimensional space, a cube was added marking the
capping of the volume; rotation was added so the point of view can change
over time, allowing a full visualization of sound in 3D.

Despite of the strong sound concept, the Chordinator experience aims to

sculpt a visual structure over time: the keyboard is controlling a form in
space, not music.

Production notes

The Chordinator employs a very simple programming concept: parametric

equations, where independent equations drive different coordinate axis.
The musical note visual component is a graph plotted through time; each
coordinate is being fed by an algorithm that determines the amplitude,
pitch and phase of sound, independently. There’s an independent
module that creates the cube and another that controls the interface. The
algorithm follows trigonometric principles, where the sound vibration can
be expressed as a sine wave, with defined amplitude, length and period.

The 3D visualization was built in Processing, and the keyboard interface

in Flash. they both work together in a browser window, allowing this
visual experience to be shared online.

77
The Harmonographer

Harmonographer interface screen

The harmonograph was an invention built and sold in the nineteen

century and it consisted in pendulums attached to a pen; the pendulum
movement set the pen coordinates, drawing the resultant of two (or
three) harmonic motions. The final drawing is a frequency interval, like
the Lissajous experiment; the pendulums also had the nice feature of
simulate the damping of a musical note, the slow decay to silence, thanks
to the mechanic energy that is dissipated within the motion. Another
feature of the harmonograph was the use of rotary motion; it added a
new dimension and perspective to the visualization. The frequencies were
more dimensional, more representative of their physical nature in space
han the Lissajous projection.

78
 an original Harmonograph

There was a strong relation between musical harmony and visual

harmony in the interval drawings of the Harmonograph; perfect intervals
were symmetrical and visually stable; numerical approximations in the
frequencies produced more unbalanced visuals, with a more intricate
nature.

The Harmonographer project is the translation of the physical device to

the computer; it has the feature of dialing in the musical notes, obtaining
the correct intervals. It also simulates the musical damping, and the
phase oscillation.

The images produced by the Harmonographer should be understood as

drawn from the outer edges (maximum volume) to the center (silence)
- the path of decay. The constant rotation of the drawing has a purpose:
show the interval at all possible sound coordination points, and its
circular motion comes from the phase value. The phase is a cyclic event (it
repeats according to the length of a wave), so there’s a constant rotation
impression to the coordinates.

The sketch also features acceleration from one interval to another

and variable interval change speed; the function of both is to induce a
smooth transition between different visuals, letting the user know what
is happening from one note to another – or just fly through all possi-
bilities.

79
A “Portamento” button is available to turn on the infinite sliding between
notes; in the portamento mode, the interval grows eternally, and the user
can keep on watching and observing how the intervals grow and their
visual elements change with the growth.

The main objective of the Harmonographer is let the user explore the
interval relations through their visual representation; or simply explore
the visual patterns and their mandala-like formations, tweaking the
possible graphical formations that the algorithm can build.

A special version of the Harmonographer was also built, that allows the
user to export the drawing as a vector file, that can be used in any graphic
application or printed. The user can also save the drawings as bitmaps,
and create video animations.

Production Notes

The coding behind the harmonographer is based on parametric equations,

independent plot instructions for x and y values. The code is slightly
more complicated than the Chordinator, since there are formulas for the
damping - that use the harmonic damping formula from physics books.
The damping was implemented as an algorithm, and the phase rotary
motion was added as well - using again the trigonometric circle concept.
The code has separate sets of acceleration and portamento functions,
and all the interface commands function, along with the Flash virtual-
keyboard element.

80
The Near Dorian Experience

Near Dorian Experience frame still

How can a musical piece be visualized through the harmonograph?

To answer this question, a video called The Near Dorian Experience was
developed, visualizing an original music composition.

The musical soundtrack intentions for the Near Dorian were to work
with few and strong musical intervals, and create a hypnotic trance of
repetition - but at the same time calm and meditative. There’s one guitar
with a mellow, woody tone and another with a twang slide sound. The
percussion is nothing more than a triangle being hit twice in the music
and the base sound is one single chord being played in a Hammond
organ, oscillating through rotating speakers. The intervals explored in
the song were pre selected through their visuals – the interesting visuals
had a granted place. The video has layers for the different instruments
and their visuals; color was added, in the form of blue tones denoting
the cold electric resonance, and yellow/orange denoting the warmness
of the guitar strings. The white denotes the purity of the bell sound, the

81
moment of illumination. Of course, this description of the soundtrack is
from a author standpoint, and the actual experience of musical meanings
may vary.

The piece shows the coordination between aural and visual intervals. The
title, Dorian, is a musical mode of the natural musical scale, known by its
simplicity. Since the song escapes the scale in a part, and the visual escapes
the simplicity due to the overlaying, it’s a near Dorian experience.

This project has the conceptual importance of being a completely

authorial project in new media – the author developed the expression
in all domains: visual, temporal, musical, and logical. All the pieces were
original and neither contained parts from other sources or projects.
Beyond that, it has the proof of possibility to develop a concept in the all
domains at the same time.

A software application was created to achieve the visual of the video,

and it couldn’t be done in another way; even if a physical harmonograph
was constructed, for example, it wouldn’t animate the phase variations
of rotating speakers - there are features that can only be achieved in a
logical medium.

The sound in space project was essentially pseudo-scientific; it had sci-

entific fundaments, but embraced creative freedom to show a personal
view of the spatial information of music. The personal view was expressed
through a computer program that helped the development of a song and a
video - the freedom of articulation came through algorithms.

82
83
Form Systems

A visual composition is formed by individual elements; these elements

interact. The study of how and why the elements in a composition interact
is the subject of visual language.

Many artists and writers published their research on the nature and prop-
erties of visual language, and one of these authors is Karl Gerstner, Swiss
painter and graphic designer. Gerstner has a remarkable point of view
regarding form and color – according to him, they’re body and spirit.
Gerstner considered math “an aid to ultimate master of one’s medium”.
His most notable characteristic is the usage of structure as a mean of
expression; many of Gerstner art pieces were not fixed compositions,
they were fixed formulas where the components could be interchanged.
Although Gerstner didn’t employed computers, he created artistic algo-
rithms; his pictures are variation machines. This computational spirit is
the inspiration for the series of form systems experiments in which the
nature of variation is explored.

Gerstner work is the inspiration for a series of experiments, where

concepts of the computational medium that connect with the idea of form
as a system are explored.

The Monkey+Gerstner and No_More_Mondrian

A good example of Gerstner’s art is the Carro 64 piece. It consists in

64 aluminum blocks, in 16 colors (a 16 step color gradient) that form a
“do-it- yourself” composition: the blocks can be arranged in any way. The
arrangements can follow symmetrical operations or be random; there are
trillions of possible combinations in the random mode, but the number
of options is smaller when compositing laws are inserted, forming deter-

85
 Carro 64 advertisement

mined color patterns.

As an experiment, I rebuilt Carro 64 as a java sketch – where the compos-

iting possibilities would be alternated but never repeated. When trans-
lated to the electronic medium, the blocks location exchange is effortless.
The canvas becomes an almost infinite art generator, rebuilding the color
blocks in a different configuration on each determined time interval.

Are the color compositions interesting? Eventually. Being random, there’s

a random possibility that an interesting or visually attractive composition

86
the original Carro 64, and one of its random computer recreations

will emerge. An interesting experiment was built on the web by the

software company AardAsNails, called Monkey Shakespeare Simulator1.
Launched in 2003, it proposes to prove the mathematical infinite monkey
theorem, by the use of bulk grid computing and random letter string gen-
eration – a monkey simulator, that would run on each user computer.

“If you have enough monkeys

banging randomly on typewriters,
they will eventually type the works
of William Shakespeare.”

The record so far is 24 letters from King Henry II – and it took only
2,737,850 million billion billion billion monkey-years to be completed.
“Eventually” is a tricky word that tends to the impossible. Maybe that’s
why Gerstner, even though creating an infinite art form, exposed configu-
rations with carefully planned symmetric harmonies that required plenty
of artistic sensibility to be achieved.

The point of this experiment is to explore the role of authorship in the

so called generative design; the absence of an author’s guidance creates
a monkey situation for the machine. There’s value on the concept of an
eternal change art canvas, but the isolated result of the variation is not
guaranteed. In a form system, the author can create rules, directions
and behaviors that will channel the creation interesting levels, instead of
abandoning the visual dimension to its own random generative luck.
1 http://user.tninet.se/~ecf599g/aardasnails/java/Monkey/webpages/index.html#results
An experiment with real monkeys was also conducted with hilarious results at
http://www.vivaria.net/experiments/notes/publication/NOTES_EN.pdf

87
 generative design processor

The other side of a generative form system occurs when a complete, vast
set of rules is laid to control the unpredictable randomness. The uncertain
composition is tamed, but the absence of the monkey factor can also
create banal pieces that always look the same.

The predictability usually occurs on systems designed to mimic an

abstract process. During the early days of graphic computing, Michael
Noll developed experiments where a computer was programmed to
mimic Mondrian’s Composition with lines. Noll’s conclusion? Viewers of
his experiment could not tell whether the painting was created by the
human Mondrian or the electronic Mondrian.

Recreating Noll’s experiment in Java has brought me some interesting

conclusions; first, it’s complex to mimic a human artistic behavior.
Second, it’s really easy to create a sketch that draws geometric forms in
random patterns, similar to many abstract paintings – but no abstract
painting is like that. Even simple geometric paintings deal with a complex
mental process of decision making: should a line be on the left or right?
Should a square be big, small or medium? What will be the horizontal to
vertical lines proportion?

To answer these questions, rules need to be written in the code. Rules that
determine average size, average proportion, placement limits, so on, so
forth. Even a minimalist painting mimic needs a great amount of rules;
the more rules, the closer the generated image will be to the original – but
it will also be more predictable. The image won’t bring new, interesting
formations that could exist outside the boundaries of a certain rule.
Any painting can have an infinite set of rules – they can grow in com-
plexity, more the rules, less the randomness, more faithful to the original

88
artwork. For the No_More_Mondrian sketch only a limited set of rules
were written, a number enough to resemble the original experiment, and
the sketch runs forever always trying to add a new layer of “ruled” shapes.
Dull, as expected.

clash of generations: the original Noll experiment and the actual Java recreation.

The Squircle

The key to creating generating form systems is striking a balance between

rigid rules and anarchic randomness.

The Squircle is a small experiment toward finding this balance. Based on

Gerstner’s Color-Form Continuum, in which he explored transitory se-
quential states between different shapes and colors, the Squircle explores
all the shapes that lie between a circle and a square and the visual compo-
sitions that derive from them.

The Squircle was built using the formula of Bézier curves - a common
tool for Postscript, CAD and vector illustration applications. Bézier curve
is a parametric cubic equation that plots a path with initial and final
points influenced by control points along the way. The control points can
be tweaked to change the overall shape of the curve. By coding a closed
shape with symmetrical control point changes, the result is a regular
polygon whose sides bend and morph in all possible positions - the

89
 screen captures
from The Squircle
and Gerstner’s
Color Form
structure model

90
square and the circle are the two extremes of the possible forms. Instead
of copying a Gerstner piece or simply creating all possible shapes in a
random fashion, the Squircle uses an oscillatory pulse that changes the
control points position, and the amplitude of the change can be set by the
mouse movement. The sole purpose of the piece is for the user to enjoy
creating forms with symmetrical aspect and experiencing the possibilities
between two basic geometric shapes.

Interactivity is a useful tool when there’s a quest to create an endless form

system that is not random; user interactivity determines a composition in
the Squircle, that is a unique and endless, but not random.

The Anaconda Paint

In another experiment, Anaconda Paint, the user builds a volumed form

according to mouse coordinates. The visual composition is obtained by
varying the color and position of a circle, with either a dented or plain
stroke texture. The illusion of a dimensional painting is created in a flat
surface by overlapping circles through time. To enhance the organic
aspect of the composition, the shape moves with acceleration toward the
cursor, avoiding abrupt movements.

It looks better in motion.

91
The “trick” that makes the Anaconda Paint interesting is the color shifting;
all the colors in the screen are constantly changing, in a cyclic way. Each
circle is assigned with a different initial color – and when all the screen
colors cycle, the illusion of volume is reinforced.

The Anaconda visual experience is supposed to be fun to play with, letting

the user create endless snake figures. This piece is completely different
from the Monkey+Gerstner, but both have the same concept: endless
compositions, based on a fixed shape, with multiple instances and varied
color and location. There are innumerous ways to translate a single
concept.

The Outline Anaconda

Roto-Champ

The idea of keeping a shape fixed, but to vary its color, size, and position
over time is full of possibilities; the Roto-Champ experiment explored
another point of view on this theme. The project is based on Marcel
Duchamp Roto-reliefs, a series of gramophone discs painted with patterns
that would rotate and create unexpected kinetic results. The Roto-Champ
is an exploration of Duchamp’s concept translated into the computational
medium.

The shapes have fixed form and color, and rotate at a constant speed.
Illusions of volume and form distortion are created by the visual inter-
action of the concentric rings, just like the original piece.

92
Marcel Duchamp invites you for the Roto-relief experience

The experiment has two different approaches; the first regards the
geometric nature of the circle. The circle is a pure mathematical form,
and its presence in a plain color background has a strong abstract and
ethereal sense. I questioned how to add more physical, concrete value
to this form - and the answer was to tweak the relative size and rotation
of the rings. The values inserted were transformed according to a value
very common to Botanic: golden section . The sacred mathematical pro-
portion is found on the growth of plants, on the patterns of seeds and on
nearly every natural organic form. Both size and rotation would respond
to organic values, linking the ethereal form to a more physical dynamic.

Roto-Champ: motion blur version

“the proportion resulting from the division of a straight line into two parts so that the ratio of the whole to
the larger part is the same as the ratio of the larger part to the smaller” – Oxford English Dictionary

93
 Roto-Champ: golden proportion and standard versions in perspective.

The second approach deals with interactivity; with one mouse coordinate,
the user can change the perspective of the piece – something that would
happen in the original Duchamp piece, since the perspective would change
according to the position of the viewer. By visually cropping parts of the
concentric figure, the illusion is changed and different visual impressions
occur. The other mouse coordinate was used to displace the center of each
ring in relation of its inner sibling, generating then different frequency
patterns, and different aesthetic experiences.

94
My Friend Jong

Form systems can be taken to extremes; as in the case of “strange

attractor” visuals. This kind of visual has been explored by a number of
designers and scientists, and it is known by algorithms that plot points
with an apparent random, chaotic, order but that “gravitate” around
certain fixed values (hence the name attractor), generating interesting
visuals. My own recreation of a strange attractor used the Peter de Jong
algorithm, written in 1991 by Paul Bourke. It was chosen because of its
simplicity - a small mathematical abstraction that can lead to complex,
visually rich results. The Jong algorithm simply subtracts a cosine from a
sine value, in both screen axis (x and y).

xn+1 = sin(a*yn) - cos(b*xn)

yn+1 = sin(c*xn) - cos(d*yn)

Peter de Jong - algorithm by Paul Bourke.

The trick is: given a starting point, the sine and cosine values are calcu-
lated based on the previous operation result, in a loop. This loop is called
iteration, and the number of iterations can be predefined – ranging from
1 pair of coordinates to billions. Organic patterns emerge from the plotted
dots– the cold mathematical abstraction is transformed in a shape that
looks and behaves like organic dynamic forces, as the smoke of an incense
stick. Given a smaller (few thousand) iteration cycles, the attractor can
be animated in real time, or respond to user interaction, like mouse
movement. Attractors are systems with little visual control (they have an
unpredictable nature) but are a good playground where small changes can
be multiplied and re applied millions of times. The concept of iteration
can be used as a creative tool – there’s not much control over the form
itself, but on how the form will emerge and unfold through iteration. This
is an important abstraction on the realm of generating images through
code: sometimes, a visual can’t be grasped; it has to be tamed through the
numbers that feed its creation.

95
 My friend Jong: 150,000,000,000 iterations.
(I counted them myself to double check if there was any missing.)

96
Riley Patterns

Op art is an artistic movement that blossomed in the middle sixties,

through the work of artists like Victor Vasarely and Bridget Riley. The
main characteristic of op art is a sensorial overload; the perception is
challenged with tricky impressions of volume, color and motion. Op art is
extremely kinetic - but in a perceptual, and not physical level.

Another trademark of op art is the creative use of patterns; visual

rhythms with distortions and modulations that conduct the composition
experience. The use of patterns and modulations usually follow a defined
set of rules – a strong sense of algorithm.

Riley and Vasarely works were not computational – their creations were
set in a time when computers where scientific secrets. My intention is
to investigate how optical visual patterns can be created and modulated
through code, and explore the computer’s advantage as a variation
machine: visual ideas can be developed in endless ways, especially when
the idea has an algorithmic nature. What are the changes in op art when
it is reclaimed by the computational medium?

Baba O’ Riley

“The art of tomorrow will be a common collective treasure or it

will not be art at all.”

-Victor Vasarely, 1960

Vasarely professed the mass media possibilities at an early date. In which

ways can art be collective?

Baba O’ Riley is an attempt to answer this question, following Gerstner’s

spirit of having structure as art. Baba O’ Riley contains no composition: it
contains the structure for endless compositions. The authorship is shared:

99
the designer authors the aesthetic framework; the audience experiences
the creation of a composition from that framework. It’s a true form of
collective art – everyone can do it.

Baba o’Riley
interface screen

The user has the ability of drawing lines and circles, with different color
frequencies - always alternating black and white. Even being just a
computer application experience, the decision process on how and where
the drawing elements should be placed is similar to the physical painting
creative process. The medium is different, but the abstract thought
process is alike

The flexibility on instantaneous composition is interesting for the artist/

designer; it allows painless experimentation. The successful tryouts
can be saved, the unsuccessful discarded – and the saved ones can even
become sketches for physical paintings. One of the major difficulties of
creating physical “real world” op art creations is the technical production
of visual patterns; they require a lot of effort and planning. A computer
application can help a lot the process of planning, easing the technical
barriers and amplifying the creative freedom.

100
Development

Baba O’ Riley was developed from scratch, programmed into the

Processing environment. It involved an interesting aspect of computer
graphics: drawing a circle. Processing was chosen because it permits
great control of the screen; pixels can be drawn and manipulated directly,
feature absent in programs like Macromedia Flash.

Designers take the computational circle drawing for granted; in Photoshop,

Illustrator, Flash or even on the simple Windows Paint, there’s a tool that
allows the cursor to draw the basic figure. But how does the tool achieve
that work?

That was the starting point of the investigation. Pixels are square; the
screen is a big group of squares. How can such a round figure exist in this
harsh environment?

The first answer: it can’t. A completely perfect abstract curve cannot be

drawn to the square based screen. It’s necessary to build a function that
returns the coordinates of the pixels that better match the path of a curve.
Curve lines on a screen are approximations; thanks to the resolution, they
look acceptably good.

The principles of trigonometry grant the knowledge of discovering such

coordinates; once the radius of the circle is known, the Pythagorean
Theorem can be applied to every pixel in the x coordinate – it will
return a matching y, creating coordinate pairs – the first step for a circle
algorithm.

square pixels failing to follow

a curved path

Unfortunately, the resulting curve line is not continuous; the frequency of

y coordinate steps wears thin toward the edges – the pixels’ imprecision
appears. The second step of the algorithm is to fill those gaps – with

101
straight lines. The result is a perfect circle – built through inconstant
coordinates and straight lines, in a computational illusion. This is the
trick that Photoshop does all the time for its users.

The third aspect of the algorithm is to build the curve with the right color.
It first evaluates the present color in the x/y coordinate, and redraws
it with the opposite value – the law of high contrast is followed all the
time.

The remainer of the code deals with mouse/keyboard response and

interface parameters. The program is very simple, and although there are
much better techniques of implementing the idea, it uses the simpler ones
in an extensively commented code. Only one basic mathematical concept
was employed – proofing that just a little math is already very useful to
express ideas through code.

The Baba O’Riley project was a success in terms of audience – the public
enjoyed the online experience (2,000 unique visitors so far) and it shows
how an interactive art project can be technically simple, but visually in-
teresting at the same time.

Current; Flow

One of the first computer graphic images in history, along with Mondrian
experiments, was a recreation of Riley’s “Current” piece, built by A.
Michael Noll, called “Ninety Parallel Sinusoids with Linearly Increasing
Period”2. I decided to explore the Noll’s experiment, due to its historical
value and also due to its visual value; Current one of the most interesting
op art paintings.

Current consists in a repetition (for Riley, repetition is a mean for ampli-

fication) of waves with increasing periods (the length of each vibration),
and constant amplitude (the height of the wave). It’s a very well math-
ematically defined piece, and maybe that’s the reason with Noll chose it
for his early computer experiments.

The objective of the first experiment was to explore the multiplicity of

means to an end; Noll’s experiment was rebuilt through different com-

2
Noll’s version was actually a recreation of Plate 3, another painting from the Current series.

102
original Noll experiment

line polygons Catmull-Rom curves

Bézier curves pixels

Current: many means to one form.

103
putational techniques – different processes for a single end. The visual
sketches were built using line polygons, Catmull-Rom splines, Bézier
Curves, and finally, direct pixel manipulation. Also, the mouse coordi-
nates were mapped to tweak amplitude and wavelength settings, helping
to mark visually appealing values.

Each technique has strong and weak features. The first sketch used line
polygons, which are fast and easy to manipulate, but are not real curves
– they’re small line segments that resemble a curve. Working with
straight lines requires less processing power, but the fake nature of the
curves might become visible. The second implementation used a curve
called Catmull-Rom; it is a curve formula, where the path goes through
the control points. Also, a control point in the curve line affects only the
segment where it belongs, and not the whole curve. Catmull-Rom curves
also have easy implementation, and are fast. Closed sinusoidal shapes
designed with Catmull-Rom are difficult to be properly filled, at least
using Java fill algorithms; the fill functions have a hard time determining
which portions of the sine wave should be convex or concave to apply the
correct filling.

Catmull-Rom is an appropriate solution – if the target code is not supposed

to become a print piece. This kind of curve is not native to PostScript, as
the Bézier curves are. For a truly cross-media piece, the solution would be
to use the famous Bézier lines.

It’s harder to achieve the perfect variable period sine wave using Bézier;
while the Catmull implementation was effortless, this required a lot of
planning and calculation on the control points positioning. It also required
many more control points to achieve the right smooth curve visual; on the
other hand, the sketch could be the size of the screen or the size of an
ocean liner, since the vector is size and resolution independent

Finally, the pixel implementation: the best way to avoid the difficult filling
of sinusoidal shapes is to draw the pixels directly. Instead of drawing a
sine shape and filling it, I can set the screen pixels with different colors
according to the form of a sine wave. The pixel version is not suitable for
printing – it is limited to the 72 dpi screen dimension – but it works fine
on screen and despite not being optimized for speed, it runs the filled sine
transformations much faster than the vector versions. Another difference

104
is that while vector drawings can have anti-aliasing functions to work on
top of them, pixel manipulation can’t – the antialiasing function has to be
written specifically to the sketch.

The Current process exploration helps to demonstrate not only that a

visual idea can be unfolded in different ways, but also that the chosen
technique influences the final aesthetic of the piece: the speed of motion,
the antialiasing algorithm, the curve rendering are unique in each
version. The concept is drawn on screen in a similar way, but the style of
the drawing is inherited from the coding techniques. In a computational
visual experience, the process matters.

The second part of the experiment dealt with the translation of a concept
from a gallery wall to the screen. Riley had very interesting ideas about
her painting – she says that her paintings aren’t about optical illusion,
but about optical reality. Riley explored the relationships between nature
and vision, and how the dynamic forms of nature could be represented
in their essence. Current has a very organic motion feeling into it, and I
wondered how to translate this feeling taking advantage of the computa-
tional medium.

The Flow sketch was the result of this exploration. It has two versions;
an interactive, where a user input – whether the mouse or any physical
device – triggers different wave settings and an automatic version, where
the waves oscillate by themselves. The sketch intends to capture the tran-
sition from a geometric state – the straight lines – to an organic behavior
resembling water currents. Dots were used instead of lines for more
delicacy, and also to imprint the idea that a current too strong becomes
noise; the dots mix themselves and the line pattern disappears.

Flow is a very simple sketch; it has few lines of code and uses line vertex
– no curves – and is even suitable for printing in any size. A simple
concept, simple implementation, interesting results.

Flow was also converted to an openGL version; openGL is a graphics

library that allows geometry to be calculated by the computer graphics
card, with increased performance in relation to the computer main
processor screen rendering. The Vanishing Flow mixes the renaissance
perspective with the sixties op art, creating a different aesthetic expe-
rience from the “flat” version.

105
 Flow: dot and perspective versions.

The op art explorations are product of my personal pursuit on exploring

and expanding this traditional art form to new possibilities; visual patterns
are great form systems that fit naturally into the creative concepts of
computation, and my role was to make the “natural inclination” become
true.

106
Logic Molecules

Database: a collection of information, stored in a retrievable system.

The database concept is the foundation of the internet today; the web is
formed by uncountable servers that store all possible categories of data,
constantly retrieved by browsers worldwide.

What does a database look like?

A database has no face; physically, it is stored in a binary mode, like any

digital information - there’s no difference between a music CD and a
complex relational database source in its physical register. Logically, it
has no face either. The interface that manages the relation between user
and database has a face – but it is a “middle man”, an agent, and it can be
simple as a blinking command prompt.

The database is intangible. The nearest a human can get to it is manipu-

lating a query program, that will dive into the structured information and
return with a completed request.

Amazon, Internet Movie Database, eBay, Craigslist, Google: the major

websites serving content and services are a shell for database querying
engines – all their power lie on the amount of information they contain,
and how the information can be retrieved. How big is Amazon? How big
is eBay? Where does it begin, and where does it end? How can a database
system be understood in a graspable form?

The “Particle Database” project investigated this issue. The intent was to
connect the idea of database to a visual materiality, and raise the identity
between databases and the physical world.

The database is a mass with different categories of information in different

amounts. It has a similar structure to physical matter: a mass built by
different molecules, from different categories in also different amounts.
When a database is queried, the relevant bits of information are retrieved
by a query agent; information is added and removed of the system as well.

109
 screenshots from the Particle Database querying system.

110
Building a parallel to physical matter, a database would be a flexible, fluid
material where the molecules could react and be attracted by different
agents.

Another feature of physical matter is density: the amount of molecules

per spatial area. A database representation would have the screen as its
total spatial area, and the density would be a key idea to represent the
amount of information populated in a system.

A potential viewer of the physical database would be able to directly

manipulate it, molding the visual data according to particular queries.
Two concepts were developed for this task; first, is the compliance to the
mechanical laws of physics. Data molecules would need to be influenced
by gravity, and their mass generate gravitational fields. Acceleration and
inertia would need to be present – by reflecting the real world, a sense
of comfort is generated in the user experience, the comfort of being ac-
quainted to physical rules like the known real world. Second, the idea of
“chemical agents”; database queries would be formulated and assume a
physical form; like a magnet or a chemical compound that would react
attracting certain types of data and repelling others. Information could
be attracted and repelled manually, allowing a sense of scale between the
amount of attracted molecules and the whole molecule universe.

The molecule/universe relationship has a clear metaphor in the compu-

tational medium: object/system. A molecule, with its components and
properties, is similar to the logical object - an element with a series of
functions and properties bound together. The concept of universe, the
multi dimensional space where molecules exist, is the same concept of a
computer system, the space where logical objects exist.

Objects are abstract data concepts; they can be modeled to contain any
visual presence on screen, and respond to any physics algorithm. Besides
an arbitrary visual representation, an object can contain actual entries
from a database.

There are scientific applications where a large number of objects with

properties need to be displayed and manipulated on screen; wind and
aerodynamics simulations, atomic reaction research, astronomical
models and many others. The name “particle system” is given to the style

111
of representing a large number of individual objects that have defined
properties, and that interact according to common rules.

The Particle Database project followed the visual style of a particle

system, having each node of information represented as a glowing pixel.
The pixels would stand grouped in a screen sized pixel sphere – the
universe, with variable density. The attraction/reaction agents would
make the particles flow around; other tools would allow certain particles
to be hidden, labeled or reveal their data content on screen. The project
was built as animated demo, non functional, where an example query is
done to envision the advantages and flexibility of direct manipulation of
data.

Coded Particles

Particle systems are the most explicit example of object oriented pro-
gramming; a logical object does not need to be visual to exist – but in a
particle field, each pixel is an object instance. The task of coding a particle
system consists in coding the behavior of a particle kind. This “kind” is
called class, the group of rules, characteristics and behaviors of an object.
The class describes all topics in the life of a particle – its appearance,
motion and dynamic responses for different stimulus caused by the user
or other particles.

The Database Field sketch is a study of how the Particle Database would
be coded; for experimental purposes, attracting and repelling behaviors
were written, and the particles behave according to a faux Brownian
motion – random in space. The sketch adds the idea of the real interac-
tivity, absent on the application demo.

interactive particle attraction/repulsion tests

112
Patterns of Time

Edwin Abbott, English writer and clergyman, published in 1884 a short

book called “Flatland”. The book is an interesting and very early exercise
into the perception of space; the human being lives in 3+1 dimensions
– 3 spatial planes and a punctum of time. The characters of Flatland live
in a 2+1 universe; thus the flatness of the title, the completely eliminated
perception of volume.

It is an interesting exercise to alter the relation between spatial and

temporal dimensions; the “Patterns of Time” project explored an inversion
on the perception of time and space; instead of 3+1 dimensions, 1+3. How
can we expand time in more dimensions, while compressing space to a
punctum? What is the spatial component of time?

The computational medium is a good support for this kind of experiment;

it allows great flexibility for experimentation, and helps the imagination
in such a entangled topic. The project proposal is an interface that allows
the compression/expansion of dimensions to occur; it does not intent to
be scientific, but poetic.

It is easy to imagine all the special dimensions compressed to a point;

timelines do that. A timeline transforms all the physical events to a
printed point; time becomes a line, dot after dot representing day after
day, year after year. But is time a line? The characters of Abbot’s Flatland
had the firm belief that a third dimension didn’t exist at all – things were
flat by nature. The visualization of time as a line of events might be stuck
in a Flatland mentality.

Pattern is a concept of many definitions, but they all agree in the idea that
a form may exist and be recurrent according to a rule – a configuration.
Patterns are everywhere in nature – the configuration of clouds, waves,
the growth of plants, structure of minerals. Common laws sew different
realms together, binding them by mathematical concepts like golden
ration and prime numbers.

opposite page: Patterns of Time Studio openGL version

115
The idea of pattern is usually employed to describe spatial or logical
events; the project idea was to extend this application to time as well
– escaping temporal Flatland.

What are the components of time?

According to a personal point of view, corroborated by the quantum

physics and approved by poetic license, time is light. Light is a periodic
wave vibration, thus time probably shares the same properties of periodic
waves - known as amplitude, wavelength and phase.

What is the amplitude, wavelength and phase of time?

���������

������

����

Phase is the point of the amplitude at which a wave begin. Applied to

time, phase should be neutral – time starts at the beginning of time (a
recursive tautology). Amplitude and wavelength are related to the notion
of facts.

What marks the passage of time? Facts1. The happening of facts in the
physical world is a beacon for proofing the time flow.

Each fact in time is unique2; but facts share similar configurations. Facts
form patterns; there’s a summer in every year – it’s a predictable event.
Each summer is unique – but events that characterize the fact “summer”
happen every year.

In different scales, all facts have patterns of occurrence – some of them

are obvious, other subtle.

The facts in time are points in the amplitude wave. Each fact follows a
pattern of occurrence, so each fact has its own frequency, wavelength.

1
“The world is the totality of facts” – Wittgenstein, in Tractatus
2
“Upon those who step into the same rivers different and ever different waters flow down.” - Heraclites

116
How to represent the concept of patterns in time visually?

The developed model of time used the line as a starting metaphor, added
with the pattern idea. The facts have a frequency of occurrence, so they
can have a cyclic nature. The end of each cycle does not mark the return
to the beginning; the line of time continues to the future. So the cycle is
not a closed circle – it is a line with a coil appearance.

Different patterns have different frequencies; bigger patterns encap-

sulate smaller ones – like an year encapsulate 365 days. So the coil is not
absolute – it is formed by another coil; which is formed by another and
other in an infinite progression.

The computer is a helper to achieve such concept in a visual way; the

initial project was done after research on coil algorithms, and how could
they be applied. A recursive coil model was developed in 3D, and a Flash
demonstration of the concepts, was built around it. The “Location of
Time” is a demonstration of the concepts of time and space, and it was
projected to be the initial point of a functional interface.

Patterns of Time Studio: interactive online version

117
 Patterns of Time: concept demo screens

118
Patterns of Time Studio: openGL version

The interactive version of the Location of Time was partially coded – at

the cost of much sweat – since the concept is very challenging mathemati-
cally. Infinite levels of recursiveness in a three dimensional structure is
not exactly the everyday problem solving task of a designer. The first levels
of time recursion can be explored through different points of view, and
different time line compression. Technically, the code describes a big three
dimensional polygon with variable sides, and variable vertex frequency.
The user can tweak and experiment on every possible parameter of the
timeline and the code is suitable for screen or print, in any given size.

119
120
Findings

As a result of creative process, the product of design has its structure

formed by concepts, form and medium. Concepts are ideas, consolidation
of the human imagination. The concepts become real through form (the
aesthetic faculties). And form needs the support of a medium, wherein it
can be expressed.

concepts

form

medium

Noteworthy design work contains a sense of equilibrium in those com-

ponents. Such work can usually be characterized as based on strong
conceptual thinking, executed by a pleasing aesthetical form within full
employment of the medium.

concepts

form

medium

time

equilibrium of the creative components

We in live in a moment in time marked by the proliferation of the personal

computer and the fast growth of the internet - a computer network as a
medium. The computer, as a vast new territory for design, is offering an
interactive and dynamic space for creation.

121
At this moment, the focus on computer as media seems to have outrun
the concepts in design creativity.

concepts

form

medium

time

media seems to have outrun the concepts in design creativity

The focus on media innovation may have eclipsed the role of concepts
and form in creation. Instead of being based on ideas, design solutions
became based on the medium; if something can move, move it; if
something can blink, let it blink. There are innumerous examples of
senseless employment of computational strengths that confirm Henry D.
Thoreau Arcadian quote “Our inventions are wont to be pretty toys, […]
they are but improved means to an unimproved end”.

A hundred years ago, a similar event occurred: the advent of cinema.

A new medium was introduced; a new technology was available for
expression. But there were neither an aesthetic nor language of cinema;
a new vocabulary needed to be built, where time was a plastic element
that could be manipulated through montage. The cinematic medium
component leaped ahead of available creative forms and concepts, and
only after years of worldwide experimentation a proper language of the
motion pictures emerged.

The concepts of computation should be taken and used to build a new

vocabulary of graphic design, where the computing logical plane shall be
seamlessly welded to the visual, aural, temporal and haptical dimensions.
The incorporation of computation is the key for conceptual thinking
to reclaim its leadership in design creativity, bringing reasoning and
meaning to interactive experiences.

122
The scientific, mathematical approach of computation and the whimsical
nature of art are not truly opposites, and have already met numerous
times through history. We are facing a new instance of this integration,
where algorithms are agents of translation from concept to form, from
form to medium.

Designers confront the challenge of learning the praxis of computation;

creative freedom in new media can only be achieved by knowing how to
deal with the transformation machine. This knowledge does not imply
that designers should become computer scientists, but that designers
should become wiser.

The visual expression through code is not – and should not – be hype,
a trendy toy of ephemeral design. Neither should it be an obligation, a
mandatory solution to all communication questions. The visual expression
through code should be considered as a current stage of the evolution in
design which is a process where different media can coexist.

The articulation of visual experiences through algorithm is the compre-

hension of new media design as an integrated whole: when conceptual
and technical barriers are tore down, creativity can be expressed simulta-
neously in all dimensions.

Creativity is borderless.

123
Bibliography

Arnheim, Rudolph. Art And Visual Perception.

Perception University of California
Press, 1974.

Ashton, Anthony. Harmonograph

Harmonograph. Walker, 2003.

Barthes, Roland. Camera Lucida. Hill and Wang, 1982.

Bense, Max. Estetica. Perspectiva, 1975.

CAVS-MIT. Gyorgy Kepes – The MIT years.

years MIT Press, 1978.

Chediak, Almir. Harmonia & Improvisação.

Improvisação Lumiar, 1986.

Dondis, Donis. A primer of visual literacy.

literacy MIT Press, 1973.

Eisenstein, Sergei. The Film Form. Harcourt, 1969.

Eisenstein, Sergei. The Film Sense. Harcourt, 1969.

Gerstner, Karl. The Spirit of Colors.

Colors MIT Press, 1981.

Gerstner, Karl. The Forms of Color. MIT Press, 1986.

Kandinsky, Wassily. Concerning the Spiritual in Art.

Art Dover, 1977.

Kandinsky, Wassily. Point and Line to Plane. Dover, 1979.

Kepes, Gyorgy. Language of Vision.

Vision Paul Theobald, 1944.

Kepes, Gyorgy et al. Language + Vision.

Vision George Braziller, 1966

Kudielka, Robert. Bridget Riley.

Riley British Council, 1978.

Maeda, John. Maeda and Media. Thames & Hudson, 2000.

Munari, Bruno. Design e Comunicação

ção Visual.
Visual Martins Fontes, 1968.

Nachmanovitch, Stephen. Free Play.

Play Tarcher, 1991.

Toben, Bob. Space Time and Beyond.

Beyond Bantam, 1983.

Wardrip-Fruin, Noah and Nick Norton. The New Media Reader. MIT
Press, 2003.

Wurman, Richard Saul. Information Anxiety I.

I Bantam, 1990.

125
127
The following is my first interactive object, developed at age of 5.
The procedure is simple:

1) turn off all the lights

2) close your eyes and position a camera flash right in front
of them
3) spin your boy, a lot
4) when it’s getting almost impossible to spin, fire the flash as
many times as possible.

I can describe the effects: colored circles, patterns, floating around.

There’s a scientific explanation, but it doesn’t really matter.