Sensors & Interactive Music 3/12/24, 8:06 PM

Sensors, interaction & performance

- creative uses of technology in live performance -

[February 12, 2002: updates pending...]

Hardware

for connecting the physical world and the computer:

Purpose-built sensors-to-MIDI units

Microcontrollers
Sensors

Software

for real-time DSP, and interactive composition and performance:

Macintosh

Institutions researching live, interactive and physical sensing applications.

My involvement in this: the "funny-fiddle" project.

Performing the studio: "liveness" and aesthetics in the digital studio.

Disclaimer

Hardware
Purpose-built units for turning physical-world sensor data into MIDI:

http://www.nealfarwell.co.uk/sensors_interactive.html Page 1 of 8
Sensors & Interactive Music 3/12/24, 8:06 PM

MIDI Brain Low-cost self-assembly kit for sensor-to-MIDI conversion.

AtoMIC (via "Products and Services"... "Software"...) Flexible and configurable converter box
from IRCAM. Has 32 x 8-bit analog-to-digital, 8 digital in and out lines, 4 MIDI-out ports.
~$800

I-Cube from Infusion Systems. Popular and powerful, the only unit I've found to offer multiple
12-bit ADC channels, along with digital I/O lines, flexible MIDI, and control objects for MAX.
I'm told it's not so rugged, but at $615 nor is it a throwaway item. Infusion also sell a wide range
of (pricey) sensors ready to plug and play.

Sensorlab from STEIM. Multiple a-to-d channels (8-bit), digital in/out lines, drivers for
ultrasonic distance measurement. Rugged, designed to be worn on stage, with a single light tether
cord to its power supply and your MIDI gear. Its own scripting language on the Mac.

Commercial "control surfaces", fader boxes, drum triggers: a popular starting point is to modify
the Peavey PC1600x as it has multiple analogue faders, switches and flexible MIDI options.

JL Cooper make various control devices. The FM3 "Faderbaby" (discontinued?) is a $99
miniature fader box that can be hand-held. I've modified one to add two joysticks.

The ADB I/O from Beehive Technologies Inc. has 4 x 8-bit ADC and 4 relay outputs, and
connects to the Apple Desktop Bus (ie not directly MIDI). Obsolete, but may still be obtainable
(Beehive is no more). Drivers for MAX, Applescript, Director 7 and others. Low sample rates
only. ~$200

Purpose-built units abused, requiring more hardware/electronics intervention:

Redundant synthesizers / cheap new computer-music keyboards ($0 - $100): for applications that
require multiple switches rather than analogue conversion, remove the "brain" - often just a small
circuit board - from a MIDI keyboard. It may also offer a few analogue channels. Keyboards with
no velocity sensing are easier to wire, just needing a push-to-make switch (and maybe a diode
per switch, to replicate the logic of the keyboard matrix). Velocity-sensing keyboards are trickier:
need break-before-make changeover switches, with a suitable transition time, but potentially rich
results. With a small amount of external electronics, can do pulse-duration measurement.

Redundant computer keyboard (<$10): as above - unsolder the switches, decipher the key matrix,
saw off the rest of the PCB from the "brain", re-use the switches in the target application. Since
this will probably be daisy-chained with the regular keyboard (eg on Macintosh ADB) it's best to

http://www.nealfarwell.co.uk/sensors_interactive.html Page 2 of 8
Sensors & Interactive Music 3/12/24, 8:06 PM

wire switch positions that will not interfere with your normal keyboard activity during the
development-cycle. A nice side-effect is that you can test the "responding" software by pressing
keys on the regular keyboard.

back to top

Microcontrollers

If your sensor data will be processed in the computer before being output to synthesizers (or
video control or whatever), the data doesn't have to arrive as MIDI. Microcontrollers, while not
specifically MIDI-oriented, are tiny self-contained computers that typically offer digital I/O lines,
often ADC or DAC, serial interface to communicate with host (or used free-standing) and non-
volatile program storage. Ubiquitous in industry, washing mashines, etc. they're cheap ($30 and
up) and small, so you build them into projects without needing to re-use. Often good price breaks
on quantity purchase.

Programming requirements vary, as do the learning curve and investment in development tools.
Some need significant hardware and software ability; others don't. If you're in an academic
institution, talk to your engineering/physics/computing colleagues about their experience with
"imbedded systems".

A selection of the many types available:

PicStic, DOMINO and AnswerMAN from Micromint.

AnswerMAN Junior: serial to 57600 bps, 4x8 bit ADC, 8 bidirectional digital lines.
Senior version adds 2x12 bit ADC/DAC and better serial line drivers. Looks
promising for the non-programmer - communication is by "query", "set" and "reply"
strings of ASCII characters. Should be easy via MAX serial object. From $49.

The PicStic comes in various flavours; one is pin- and code- compatible with the
BASIC Stamp 1, but gives access to interrupts and timer. Micromint claim their
BASIC is 15 times faster (because compiled rather than interpreted); you can also use
assembler, C, or a mixture. The compiler provides serial routines up to 9600 bps - my
guess is you could code MIDI-speed routines. Variants include 2x12 bit ADC and
realtime clock/calender. Flexible, but investment in developer environment.
Physically tiny. Cost from $29.

The DOMINO is a general microcontroller running a floating-point BASIC interpreter

http://www.nealfarwell.co.uk/sensors_interactive.html Page 3 of 8
Sensors & Interactive Music 3/12/24, 8:06 PM

(which also allows assembly language routines). Serial interface to 19200 bps, 2x12
bit ADC, I2C bus. From $99.

back to top

The following are outline notes only. More critical assessment in due course...

Basic Stamp from Parallax Inc.

Popular with non-specialist-engineers; runs interpreted tokenized BASIC but no

assembler; B-Stamp 2 runs ~4000 instructions/sec; PC dev environment, or Mac
running Soft PC or Soft Windows 2.0; PIC16C56 (BS1); PIC16C57 (BS2); timer is
used internally; no external interrupts; BS2 can hold 5-600 lines of PBASIC2 code;
efficient IO commands; 16-bit integer arithmetic; BS2 "can handle MIDI serial
speed"; comparing BS2 schematic & PIC16C57 data sheet indicates serial port (eg for
MIDI) is handled by software on general-purpose IO pins (so nothing else happens
while serial bytes are being sent). Can be a tidy solution if you don't have a huge
volume of data throughput.

BASIC Tiger / Tiny Tiger from Wilke Technology GmbH

Modules, look to have a lot of power for the price; Multitasking - useful or a liability?
12-bit ADC "with oversampling"; Windows dev environment; TT from $59 - dev kit
$149; BT from $89 - dev kit $299

BasicX by NetMedia Inc.

Single chip, like core of BASIC Tiger; multitasking; cheap - dev kit $50 - includes
chip, etc; single chip $40, $20 for 10-off; PC dev environment, needs parallel port;
has "two high speed bidirectional buffered serial ports"; 65,000 lines of code per sec.;
small progs don't need external EEPROM or RAM

RP3

Larger circuit board; Nice range of features but $400 for RPC-330; BASIC, C,
assembler; library of drivers; 8x12bit adc, serials, 2 fast counters, etc.

FIRECARD from Fire, Wind & Rain

Pcb, but small; good features, 8x12bit adc, serials, floating-point BASIC etc; flash

http://www.nealfarwell.co.uk/sensors_interactive.html Page 4 of 8
Sensors & Interactive Music 3/12/24, 8:06 PM

memory cards; price? serial speed?

AVR series from ATMEL

Powerful RISC-based chips, with wide range of on-chip peripherals. Heavy-duty data
sheets and development environmnet. Not for the faint hearted?

Some Macintosh-based development tools can be found at MacRobotics and at Francis J. Deck's
site (for PIC programming); DesignWorks from Capilano Computing (circuit schematic capture);
and Douglas Electronics for PCB design. PCB prototype manufacturers include Douglas, AGCO
Printed Circuitry Inc., ECD PCB Express and Alberta Printed Circuits Ltd.. Newark Electronics,
Jameco Electronics and Digi-Key are suppliers of electronics parts in the USA.

I found the magazine Circuit Cellar INK a useful source here (in its printed form). Thanks also to
colleagues on the MAX-list.

Disclaimer

back to top

Sensors

A directory of sources for input technologies prepared by Bill Buxton.

I-Cube ready-rolled.

IRCAM - see below under Institutions

Robotics sites are a good source of sensor ideas - try Acroname.

Sensors designed for home security, such as pressure mats and motion detectors, are available
from many mail-order electronics suppliers, Radio Shack, etc.

Software for the Apple Mac:

Max/MSP is "an interactive real-time graphic programming environment for multimedia," a
toolkit for building anything from quick-and-dirty MIDI patchers to complex, polished DSP
systems. Max/MSP is widely used by performers, composers, sound designers, researchers and
educators. Smooth handling of MIDI data is central to Max (a non-audio version is available),
http://www.nealfarwell.co.uk/sensors_interactive.html Page 5 of 8
Sensors & Interactive Music 3/12/24, 8:06 PM

but it can do much more. You can also build license-free standalone applications.

Max/MSP users maintain (a) strong community spirit(s). The software's makers Cycling '74
actively support this with a community page, including links to discussion lists and an extensive
Max/MSP Annotated Resource Guide. (There is some overlap here with the material of this
page.)

Highlight include a set of tutorials by Peter Elsea.

UnMAX maintains a download resource of Max/MSP/nato objects, patches and add-ons.

Extra MAX patches Useful add-ons. IRCAM's site includes excellent free/shareware MAX
objects to add AIFF file playback, speech control etc. to the basic MAX environment. (A good
half-way step to the non-free MSP...)

For video manipulation, nato.0+55 is a powerful and evolving extension to the Max environment.
The author has a unique net identity/image.

SuperCollider "A real time audio synthesis programming language." Not for the beginner: a
C++/SmallTalk -like text based language. But very slick, very fast; clean audio (eg anti-aliased
oscillators, sorely lacking from many r-t synthesis applications); easy graphic interface building
and MIDI mapping; and the new version multitasks with other application (including MIDI and
audio). Program development is fast when you get used to it.

Also check STEIM's real-time performance software.

back to top

Institutions
The MIT Media Lab is a high-profile, high-budget academic research institute. Cutting-edge
R&D in cognitive, perceptual, algorithmic etc. engineering seems to take precedence over
aesthetic production.

The University of Surrey, UK, had recently established the Institute for New Media Performance
Research within its School of Performing Arts. The focus is on making collaborative and
interactive music/dance/theatre, supported by a framework of academic critique.

IRCAM in Paris has an excellent site on Gesture Research in Music, run by Marcelo M.
Wanderley. This draws together research and resources across the aesthetic-technological-critical
http://www.nealfarwell.co.uk/sensors_interactive.html Page 6 of 8
Sensors & Interactive Music 3/12/24, 8:06 PM

spectrum.

STEIM The STudio for Electro Instrumental Music in Amsterdam, the Netherlands. New (art-
)music performance technologies, R&D, artist residencies, performances. They sell live-
performance oriented software for sampling, video, MIDI.

back to top

My involvement: the "funny-fiddle" project

This material was drawn together as part of an ongoing

personal project. I have developed a violin-like control
instrument, putting a live solo performer at the focus of
the electronic sound-world. I sought to draw on my
training as a violinist, without the sounds produced
having any necessary "string" component. (This is a
problem for me with the instrument-plus-sensors
approach of, for example, the Zeta MIDI violin.) Violin-
like gestural activity is translated not only into what sounds happen when, but also where in the
performance space they sound. Present development of the instrument is focused on deepening
its sensitivity to the control of micro-gesture that a classical training yields. (It'll get four strings,
too.)

The compositional challenge is to make a rich musical work that integrates vitally with the
performance showmanship. A first work for the instrument, Gipsy fugue (1996-7), was very well
received in its two concert performances at UEA and was featured in a network television
documentary for the Open University.

back to top

Performing the studio

This discussion of tools has implicitly focused on situations where live performance is
"expected" or is the foreground activity. Bringing these tools into the computer studio opens a
space of speculative activity informed by physical gesture.

Here is a small practical example: many digital mixing desks send out MIDI for movements of
their faders, knobs and push-buttons. If you can temporarily spare the mix channels, use the desk
as a generalized control surface. Map the faders etc to parameters of a software or hardware

http://www.nealfarwell.co.uk/sensors_interactive.html Page 7 of 8
Sensors & Interactive Music 3/12/24, 8:06 PM

sound-processor or synthesizer (not just level, pan, EQ). Improvise sonic trajectories. Listen in
real time. What happens when you think about the mapping of controllers to process parameters?
What happens if you think about the mapping of physical gesture to sonic result (and ignore the
process parameters)? Rehearse, vary, riff, counterpoint?

An exploration of the aesthetic inflections, transplantations and transgressions implied here is the
rich subject of research, inside and outside the academy, by practitioners and by theorists. Here,
in keeping with the spirit of this page, I offer the practical challenge.

The usual disclaimers

I have no financial interest in any of the companies mentioned (other than my own search for
cost-effective tools). Some of these tools I've used, some I scanned manufacturer's data sheets,
some may be hear-say. Prices and specs may change, etc., etc. Suggestions that involve
"engineering" assume you have enough knowledge to look after your own and your equipment's
physical safety. Any use of this information or of the products described is entirely at your own
risk.

Please help me refine and expand this information. Email your responses.

back to top

© Neal Farwell. Last updated August 2001.

this page: Sensors, interaction & performance

home || introduction || music links || other links

list of works || about this web site || mail me

http://www.nealfarwell.co.uk/sensors_interactive.html Page 8 of 8