Jean-François Charles

1 Route de Plampéry
A Tutorial on Spectral
74470 Vailly, France
[email protected] Sound Processing Using
Max/MSP and Jitter

For computer musicians, sound processing in the This article is intended as both a presentation of
frequency domain is an important and widely used the potential of manipulating spectral sound data
technique. Two particular frequency-domain tools as matrices and a tutorial for musicians who want
of great importance for the composer are the phase to implement such effects in the Max/MSP/Jitter
vocoder and the sonogram. The phase vocoder, an environment. Throughout the article, I consider
analysis-resynthesis tool based on a sequence of spectral analysis and synthesis as realized by the
overlapping short-time Fourier transforms, helps Fast Fourier Transform (FFT) and Inverse-FFT
perform a variety of sound modifications, from time algorithms. I assume a familiarity with the FFT
stretching to arbitrary control of energy distribution (Roads 1995) and the phase vocoder (Dolson 1986).
through frequency space. The sonogram, a graphical To make the most of the examples, a familiarity
representation of a sound’s spectrum, offers com- with the Max/MSP environment is necessary, and
posers more readable frequency information than a a basic knowledge of the Jitter extension may be
time-domain waveform. helpful.
Such tools make graphical sound synthesis I begin with a survey of the software currently
convenient. A history of graphical sound synthesis is available for working in this domain. I then show
beyond the scope of this article, but a few important some improvements to the traditional phase vocoder
figures include Evgeny Murzin, Percy Grainger, and used in both real time and performance time.
Iannis Xenakis. In 1938, Evgeny Murzin invented a (Whereas real-time treatments are applied on a live
system to generate sound from a visible image; the sound stream, performance-time treatments are
design, based on the photo-optic sound technique transformations of sound files that are generated
used in cinematography, was implemented as the during a performance.) Finally, I present extensions
ANS synthesizer in 1958 (Kreichi 1995). Percy to the popular real-time spectral processing method
Grainger was also a pioneer with the “Free Music known as the freeze, to demonstrate that matrix
Machine” that he designed and built with Burnett processing can be useful in the context of real-time
Cross in 1952 (Lewis 1991); the device was able effects.
to generate sound from a drawing of an evolving
pitch and amplitude. In 1977, Iannis Xenakis and
associates built on these ideas when they created Spectral Sound Processing with Graphical
the famous UPIC (Unité Polyagogique Informatique Interaction
du CEMAMu; Marino, Serra, and Raczinski 1993).
In this article, I explore the domain of graphical Several dedicated software products enable graphic
spectral analysis and synthesis in real-time situa- rendering and/or editing of sounds through their
tions. The technology has evolved so that now, not sonogram. They generally do not work in real time,
only can the phase vocoder perform analysis and because a few years ago, real-time processing of com-
synthesis in real time, but composers have access to plete spectral data was not possible on computers
a new conceptual approach: spectrum modifications accessible to individual musicians. This calculation
considered as graphical processing. Nevertheless, the limitation led to the development of objects like
underlying matrix representation is still intimidat- IRCAM’s Max/MSP external iana∼, which reduces
ing to many musicians. Consequently, the musical spectral data to a set of useful descriptors (Todoroff,
potential of this technique is as yet unfulfilled. Daubresse, and Fineberg 1995). After a quick
survey of the current limitations of non-real-time
Computer Music Journal, 32:3, pp. 87–102, Fall 2008 software, we review the environments allowing FFT


c2008 Massachusetts Institute of Technology. processing and visualization in real time.

Charles 87
 Figure 1. FFT data
recorded in a stereo buffer.

Non-Real-Time Tools

AudioSculpt, a program developed by IRCAM, is

characterized by the high precision it offers as well
as the possibility to customize advanced parameters
for the FFT analysis (Bogaards, Röbel, and Rodet Real-Time Environments
2004). For instance, the user can adjust the analysis
window size to a different value than the FFT size. PureData and Max/MSP are two environments
Three automatic segmentation methods are pro- widely used by artists, composers, and researchers
vided and enable high-quality time stretching with to process sound in real time. Both enable work in
transient preservation. Other important functions the spectral domain via FFT analysis/resynthesis. I
are frequency-bin independent dynamics process- focus on Max/MSP in this article, primarily because
ing (to be used for noise removal, for instance) the MSP object pfft∼ simplifies the developer’s
and application of filters drawn on the sonogram. task by clearly separating the time domain (outside
Advanced algorithms are available, including funda- the pfft∼ patcher) from the frequency domain
mental frequency estimation, calculation of spectral (inside the patcher). When teaching, I find that the
envelopes, and partial tracking. Synthesis may be graphical border between the time and frequency
realized in real time with sequenced values of pa- domains facilitates the general understanding of
rameters; these can, however, not be altered “on the spectral processing.
fly.” The application with graphical user interface Using FFT inside Max/MSP may be seen as
runs on Mac OS X. difficult because the FFT representation is two-
MetaSynth (Wenger and Spiegel 2005) applies dimensional (i.e., values belong to a time/frequency
graphical effects to FFT representations of sounds grid), whereas the audio stream and buffers are
before resynthesis. The user can apply graphical one-dimensional. Figure 1 shows how data can be
filters to sonograms, including displacement maps, stored when recording a spectrum into a stereo
zooming, and blurring. A set of operations involv- buffer.
ing two sonograms is also available: blend, fade The tension between the two-dimensional nature
in/out, multiply, and bin-wise cross synthesis. It of the data and the one-dimensional framework
is also possible to use formant filters. The pitfall makes coding of advanced processing patterns (e.g.,
is that MetaSynth does not give control of the visualization of a sonogram, evaluation of transients,
phase information given by the FFT: phases are and non-random modifications of spatial energy in
randomized, and the user is given the choice among the time/frequency grid) somewhat difficult. An
several modes of randomization. Although this example of treatment in this context is found in the
might produce interesting sounds, it does not offer work of Young and Lexer (2003), in which the energy
the most comprehensive range of possibilities for in graphically selected frequency regions is mapped
resynthesis. onto synthesis control parameters.
There are too many other programs in this vein The lack of multidimensional data within
to list them all, and more are being developed Max/MSP led IRCAM to develop FTM (Schnell
each year. For instance, Spear (Klingbeil 2005) fo- et al. 2005), an extension dedicated to multidimen-
cuses on partial analysis, SoundHack (Polansky sional data and initially part of the jMax project.
and Erbe 1996) has interesting algorithms for spec- More specifically, the FTM Gabor objects (Schnell
tral mutation, and Praat (Boersma and Weenink and Schwarz 2005) enable a “generalized granular
2006) is dedicated to speech processing. The lat- synthesis,” including spectral manipulations. FTM
ter two offer no interactive graphical control, attempts to operate “Faster Than Music” in that
though. the operations are not linked to the audio rate but

88 Computer Music Journal

Figure 2. Phase vocoder
design.

are done as fast as possible. FTM is intended to be data in a matrix, to transform it (via zoom and
cross-platform and is still under active development. rotation), and to play it back via the IFFT; second,
However, the environment is not widely used at the an extension of this first work, where the authors
moment, and it is not as easy to learn and use as introduce graphical transformations using a transfer
Max/MSP. matrix (Sedes, Courribet, and Thiebaut 2004).
In 2002, Cycling ’74 introduced its own mul- Both use a design similar to the one presented in
tidimensional data extension to Max/MSP: Jitter. Figure 2.
This package is primarily used to generate graphics This section endeavors to make the phase-vocoder
(including video and OpenGL rendering), but it en- technique accessible, and it presents improvements
ables more generally the manipulation of matrices in the resynthesis and introduces other approaches
inside Max/MSP. Like FTM, it performs operations to graphically based transformations.
on matrices as fast as possible. The seamless in-
tegration with Max/MSP makes implementation
of audio/video links very simple (Jones and Nevile FFT Data in Matrices
2005).
Jitter is widely used by video artists. The learning An FFT yields Cartesian values, usually translated
curve for Max users is reasonable, thanks to the to their equivalent polar representation. Working
number of high-quality tutorials available. The with Cartesian coordinates is sometimes useful for
ability to program in Javascript and Java within Max minimizing CPU usage. However, to easily achieve
is also fully available to Jitter objects. Thus, I chose the widest and most perceptively meaningful range
Jitter as a future-proof choice for implementing this of transformations before resynthesis, we will work
project. with magnitudes and phase differences (i.e., the bin-
by-bin differences between phase values in adjacent
frames).
An Extended Phase Vocoder Figure 3 shows how I store FFT data in a Jitter
matrix throughout this article. We use 32-bit
Implementing a phase vocoder in Max/MSP/Jitter floating-point numbers here, the same data type
unlocks a variety of musical treatments that would that MSP uses. The grid height is the number of
remain impossible with more traditional approaches. frequency bins, namely, half the FFT size, and its
Previous work using Jitter to process sound in the length is the number of analysis windows (also called
frequency domain include two sources: first, Luke frames in the FFT literature). It is logical to use a
Dubois’s patch jitter pvoc 2D.pat, distributed two-plane matrix: the first is used for magnitudes,
with Jitter, which shows a way to record FFT the second for phase differences.

Charles 89
Figure 3. FFT
representation of a sound
stored in a matrix.

Interacting with a Sonogram The patch in Figure 4 shows a simple architecture

for interacting with the sonogram. Equations 1
The sonogram represents the distribution of energy and 2 and part of Equation 3 are implemented inside
at different frequencies over time. To interact with patcher (or p) objects. To reduce computational
this representation of the spectrum, we must scale cost, the amplitude plane of the matrix holding
the dimension of the picture to the dimension of the the FFT data is converted to a low-resolution
matrix holding the FFT data. To map the matrix matrix before the display adjustments (i.e., the
cell coordinates to frequency and time domains, we inversion of low/high frequencies and inversion of
use the following relations. First, time position t (in black and white, because by default, 0 corresponds
seconds) is given by to black and 1 to white in Jitter). The OpenGL
implementation of the display is more efficient
WindowSize 1
t = n× × (1) and flexible, but it is beyond the scope of this
OverlapFactor sr article.
where n is the number of frames, sr is the sampling
rate (Hz), and WindowSize is given in samples.
Second, the center frequency fc (Hz) of the frequency Recording
bin m is
sr When implementing the recording of FFT data into
fc = m× (2) a matrix, we must synchronize the frame iteration
FFTSize
with the FFT bin index. The frame iteration must
Third, assuming no more than one frequency is happen precisely when the bin index leaps back to
present in each frequency bin in the analyzed signal, 0 at the beginning of a new analysis window. Luke
its value in Hz can be expressed as Dubois’s patch jitter pvoc 2D.pat contains
sr one implementation of a solution. (See the object
f = fc +
φ × (3) count∼ in [pfft∼ jit.recordfft∼.pat].)
2π × WindowSize
where
φ is the phase difference, wrapped within
the range [–π , π ] (Moore 1990). Note that with the Playback
pfft∼ object, the window size (i.e., frame size) is
the same as the FFT size. (In other words, there is Figure 5 shows a simple playback algorithm using an
no zero-padding.) IFFT. A control signal sets the current frame to read.

90 Computer Music Journal

Figure 4. Interaction with
a sonogram.

To play back the sound at the original speed, the each analysis window with an original duration of 23
control signal must be incremented by one frame at msec is stretched out to 836 msec. During synthesis
a period equal to the hop size. Hence, the frequency with the traditional phase-vocoder method, the leap
f (in Hz) of the phasor∼ object driving the frame from one frame to the following one is audible with
number is a low playback speed. This particular artifact, the
sr “frame effect,” has not received much attention
f = × PlaybackRate (4) in the literature. I will describe two methods to
N × HopSize
interpolate spectra between two recorded FFT
where N is the total number of analysis windows. frames, one of which can easily be extended to a
Here, PlaybackRate sets the apparent speed of the real-time “blurring” effect.
sound: 1 is normal, 2 is twice as fast, 0 freezes one When humans slow down the pacing of a spoken
frame, and –1 plays the sound backwards. sentence, they are bound to slow down vowels
more than consonants, some of which are physically
impossible to slow down. Similarly, musicians can
Advanced Playback naturally slow down the tempo of a phrase without
slowing down the attacks. That is why time-
In extreme time stretching, the two main artifacts stretching a sound uniformly, including transients,
of the phase vocoder are the “frame effect” and the does not sound natural. Transient preservation in the
time-stretching of transients. In my composition phase vocoder has been studied, and several efficient
Plex for solo instrument and live electronics, ten algorithms have been presented (Laroche and Dolson
seconds are recorded at the beginning of the piece 1999; Röbel 2003). Nevertheless, these propositions
and played back 36 times more slowly, spread over have not been created in a form adapted to real-time
6 minutes. Considering an FFT with a hop size of environments. I present a simple, more straightfor-
1,024 samples and a sampling rate of 44,100 Hz, ward approach that computes a transient value for

Charles 91
Figure 5. A simple player.

each frame and uses it during resynthesis. That will a frame number of 4.6, the value in a cell of the
naturally lead to an algorithm for segmentation, interpolated matrix is the sum of 40 percent of the
designed for performance time as well. value in the corresponding cell in frame 4 and 60
percent of the value in frame 5.
Removing the “Frame Effect” My second approach is a controlled stochastic
spectral synthesis. For each frame to be synthesized,
This section shows two algorithms interpolating each amplitude/phase-difference pair is copied from
intermediary spectra between two “regular” FFT the recorded FFT data, either from the corresponding
frames, thus smoothing transitions and diminishing frame or from the next frame. The probability of
the frame effect. In both cases, a floating-point picking up values in the next frame instead of the
number describes the frame to be resynthesized: for current frame is the fractional value given by the
instance, 4.5 for an interpolation halfway between user. For instance, a frame number of 4.6 results in
frames 4 and 5. a probability of 0.4 that values are copied from the
In the first method, the synthesis module con- original frame 4 and a probability of 0.6 that values
tinuously plays a one-frame spectrum stored in a are copied from the original frame 5. This random
one-column matrix. As illustrated in Figure 6, this choice is made for each frequency bin within
matrix is fed with linear interpolations between two each synthesis frame, such that two successive
frames from the recorded spectrum, computed with synthesized frames do not have the same spectral
the jit.xfade object. Values from both amplitude content, even if the specification is the same. The
and phase difference planes are interpolated. With implementation shown in Figure 7 shows a small

92 Computer Music Journal

Figure 6. Interpolated Figure 7. A smooth player.
spectrum between two
analysis windows.

Figure 6

Figure 7

Charles 93
Figure 8. Stochastic
synthesis between two
frames, blur over several
frames.

modification to the engine presented Figure 5. A computed in the low-priority thread, meaning that
signal-rate noise with amplitude from 0 to 1 is added the CPU load increases with the number of frames
to the frame number. While reading the FFT data to calculate. Second, the stochastic method offers a
matrix, jit.peek∼ truncates the floating-point built-in blurring effect, whereas achieving this effect
value to an integer giving the frame number. The with matrix interpolation would require additional
vertical synchronization is removed to enable the programming and be less responsive. In what
frame number to change at any time. Increasing the follows, I continue to develop this second method,
floating-point value in the control signal creates an because it adheres to the original phase-vocoder
increased probability of reading the next frame. architecture and is more flexible.
This method can be readily extrapolated to blur
any number of frames over time with negligible
additional cost in CPU power. Instead of adding a Transient Preservation
noise between 0 and 1, we scale it to [0, R], where
R is the blur width in frames. If C is the index of I propose a simple algorithm for performance-time
the current frame, each pair of values is then copied transient evaluation, along with a complementary
from a frame randomly chosen between frames C algorithm to play a sound at a speed proportional
and C + R, as shown in Figure 8. The musical result to the transient value. My approach consists in
is an ever-moving blur of the sound, improving the assigning to each frame a transient value, defined
quality of frozen sounds in many cases. The blur as the spectral distance between this frame and the
width can be controlled at the audio rate. preceding frame, normalized to [0, 1] over the set of
Both the interpolated frame and the stochastic frames.
playback methods produce high-quality results Several choices are possible for the measure of
during extremely slow playback, but they are the spectral distance between two frames. The
useful in different contexts. The interpolated frames Euclidean distance is the most obvious choice.
often sound more natural. Because it lacks vertical Given M frequency bins, with am,n the amplitude in
coherence, the stochastic method may sound less bin m of frame n, the Euclidean distance between
natural, but it presents other advantages. First, frames n and n – 1 is
it is completely implemented in the “perform”

 M−1
thread of Max/MSP; as a result, CPU use is constant 
regardless of the parameters, including playback tn =  (am,n − am,n−1 )2 (5)
m= 0
rate. However, the interpolated spectrum must be

94 Computer Music Journal

Figure 9. A transient value
for each frame.

In this basic formula, we can modify two parameters rate rstat (the playback rate for the most stationary
that influence quality and efficiency: first, the set of part of the sound). Given the current frame’s
descriptors out of which we calculate the distance, transient value tr, the instantaneous playback rate
and second, the distance measure itself. The set rinst is given by
of descriptors used in Equation 5 is the complete
set of frequency bins: applying Euclidean distance rinst = rstat + tr × (rtrans − rstat ) (7)
weights high frequencies as much as low ones, The musical result of a continuous transient value
which is not perceptually accurate. For a result between 0 and 1 is interesting, because it offers a
closer to human perception, we would weight the built-in protection against errors of classification
different frequencies according to the ear’s physi- as transient or stationary. This is especially useful
ology, for instance by using a logarithmic scale for in a performance-time context. A binary transient
the amplitudes and by mapping the frequency bins value could, of course, be readily implemented by a
non-linearly to a set of relevant descriptors like comparison to a threshold applied to all the values
the 24 Bark or Mel-frequency coefficients. In terms of the transient matrix with a jit.op object.
of CPU load, distance is easier to calculate over Controlling the blur width (see Figure 8) with the
24 coefficients than over the whole range of fre- transient value may be musically useful as well. The
quency bins, but the calculation of the coefficients most basic idea is to make the blur width inversely
makes the overall process more computationally proportional to the transient value, as presented
expensive. Although using the Euclidean distance in Figure 11. Similarly to the playback rate, the
makes mathematical sense, the square root is a rela- instantaneous blur width binst is given by
tively expensive calculation. To reduce computation
load, in Figure 9 I use in a rough approximation of binst = bstat + tr × (btrans − bstat ) (8)
the Euclidean distance:
where tr is the current frame transient value, bstat is

M−1
  the stationary blur width, and btrans is the transient
tn = am,n − am,n−1  (6) blur width.
m= 0

The playback part of the vocoder can use the Segmentation

transient value to drive the playback rate. In the
patch presented Figure 10, the user specifies the This section describes a performance-time method of
playback rate as a transient rate rtrans (the playback segmentation based on transient values of successive
rate for the greatest transient value) and a stationary frames. I place a marker where the spectral changes

Charles 95
Figure 10. Re-synthesizing Figure 11. Blur width
at self-adapting speed. controlled by transient
value. During the
playback of stationary
parts of the sound (left),
the blurred region is wider
than during the playback
of transients (right).

Figure 10

Figure 11

from frame to frame are greater than a given result. In the preceding section, we used Euclidean
threshold. As Figure 12 shows, frame n is marked distance or an approximation by absolute difference
if and only if its transient value is greater than or (Equations 5 and 6). Both expressions yield a useful
equal to the transient value in the preceding frame auto-adaptive playback rate, but my experience
plus a threshold h, that is, whenever is that the following expression is preferable for
ulterior segmentation with Equation 9:
tn ≥ tn−1 + h (9)

M−1
am,n
The formula used to calculate transient values ap- tn = (10)
pears to have a great influence on the segmentation am,n−1
m= 0

96 Computer Music Journal

Figure 12. Automatic
performance-time
segmentation.

Indeed, whereas Euclidean distance or absolute and evaluates expressions to an output matrix.
difference give spectrum changes a lower weight Moreover, all Jitter operators compatible with 32-bit
at locally low amplitudes than at high amplitudes, floating-point numbers are available in jit.expr.
the division in Equation 10 enables a scaling of the Phase information is important for the quality
degree of spectrum modification to the local level of of resynthesized sounds. The choice to apply a
amplitude. Musical applications include real-time treatment to amplitude and phase difference or only
leaps from one segmentation marker to another, to amplitude, or to apply a different treatment to
resulting in a meaningful shuffling of the sound. the phase-difference plane, is easily implemented
An important point to retain from this section is whether in jit.expr or with jit.pack and
that, whereas ideas may materialize first as iterative jit.unpack objects. This choice depends on the
expressions (e.g., see the summations in Equations particular situation and must generally be made
5, 6, and 10), the implementation in Jitter is reduced after experimenting.
to a small set of operations on matrices. To take In Figure 13, the expression attribute of the
full advantage of Jitter, we implement parallel second jit.expr object is [‘‘gtp(in[0].p[0]\,
equivalents to iterative formulas. 1.3)’’ ‘‘in[0].p[1]’’]. The first part of the
expression is evaluated onto plane 0 of the output
matrix; it yields the amplitude of the new sound. It
Graphically Based Transformations applies the operator gtp (which passes its argument
if its value is greater than the threshold, otherwise,
Now, we explore several ways of transforming a it yields 0) to the amplitude plane of the incoming
sound through its FFT representation stored in two- matrix with the threshold 1.3. It is a rough denoiser.
dimensional matrices. The four categories I examine (In the previous expression, in[0].p[0] is plane 0
are direct transformations, use of masks, interaction of the matrix in input 0.) The second part of the
between sounds, and “mosaicing.” expression produces on plane one the first plane of
the incoming matrix (i.e., in[0].p[1]); the phase
is kept unchanged.
Direct Transformations

The most straightforward transformation consists Masks

of the application of a matrix operation to an FFT
data matrix. In Jitter, we can use all the objects that Masks enable the gradual application of an effect
work with 32-bit floating-point numbers. A flexible to a sound, thus refining the sound quality. A
implementation of such direct transformations is different percentage of the effect can be used in
possible thanks to jit.expr, an object that parses every frequency bin and every frame. In this article,

Charles 97
Figure 13. Graphical Figure 14. Two designs to
transformations. use masks.

Figure 13

Figure 14

I limit the definition of a mask to a grid of the Figure 14 presents two possible designs. In the
same resolution as the FFT analysis of the sound, left version, the processing uses the mask to produce
with values between 0 and 1. In Jitter, that means the final result; this is memory-efficient. In the
a one-dimensional, 32-bit floating-point matrix, of right version, the original sound is mixed with the
the same dimension as the sound matrix. processed one; this allows a responsive control on
A mask can be arbitrarily generated, interpolated the mix that can be adjusted without re-calculation
from a graphical file, drawn by the user on top of the of the sound transformation.
sound sonogram, or computed from another sound
or the sound itself. For example, let us consider
a mask made of the amplitudes (or the spectral Interaction of Sounds
envelope) of one sound, mapped to [0, 1]. Multiplying
the amplitudes of a source sound with this mask is Generalized cross synthesis needs both amplitude
equivalent to applying a source-filter cross synthesis. and phase difference from both sounds. Similarly,

98 Computer Music Journal

Figure 15. Context-free
mosaicing algorithm.

interpolating and morphing sounds typically re- a simple mosaicing by spectral frame similarity,
quires amplitude and phase information from both similar to the Spectral Reanimation technique (Lyon
sounds. 2003). Before resynthesis, each FFT frame in the
It is easy to implement interpolation between target sound is replaced by the closest frame in the
sounds with jit.xfade, in the same way as database irrespective of the context. The algorithm
interpolation between frames in Figure 6. An is illustrated in Figure 15.
interesting musical effect results from blurring the Three factors influence the speed and quality of
cross-faded sound with filters such as jit.streak, the resulting sound. First, the amplitude scale might
jit.sprinkle , and jit.plur. be linear (less CPU intensive) or logarithmic (closer
As opposed to an interpolated sound, a morphing to perception). Second, the descriptors for each
between two sounds evolves from the first to the frame can take variable sizes. The set of amplitudes
second within the duration of the synthesized sound in each frequency bin is directly available, but it is
itself. Such a morphing can be implemented as an a large vector. A linear interpolation to a reduced
interpolation combined with three masks: one for number of bands, a set of analyzed features (pitch,
the disappearance of the first source sound, a second centroid, etc.), and a nonlinear interpolation to the
for the appearance of the second sound, and a third 24 Bark or Mel-frequency coefficients are other
one to apply a variable amount of blurring. possible vectors. The third factor is the distance
used to find the closest vector (Euclidean dis-
tance, or an approximation sparing the square-root
Mosaicing calculation).
My choice has been to use a logarithmic am-
Mosaicing is a subset of concatenative synthesis, plitude scale, a vector of Bark coefficients, and
where a target sound is imitated as a sequence of Euclidean distance or an approximation similar to
best-matching elements from a database (Schwarz Equation 6. The Bark coefficients, reflecting ear
2006). In the context of the phase vocoder, I provide physiology, are well suited to this task.

Charles 99
Figure 16. Real-time
stochastic freeze
implementation.

Real-Time Freeze and Beyond transition sub-patch between the matrix to freeze
(freeze8-record) and the matrix currently playing
The matrix approach to spectral treatments allows (freeze8-play) can be used to cross-fade when a
not only a variety of performance-time extensions new sound is frozen. Such sub-patches incorporate
to the phase vocoder but also improvements in first a counter to generate a number of transition
real-time processing, such as freezing a sound and matrices, and second a cross-fading object such
transforming a melody into a chord. as jit.xfade controlled by a user-drawn graph
specifying the cross-fade curve.
Real-Time Freeze

A simple way to freeze a sound in real time is Melody to Harmony

to resynthesize one spectral frame continuously. I
improve the sound quality by freezing several frames In my composition Aqua for solo double bass,
at once and then resynthesizing the complete set aquaphon, clarinet, and live electronics, arpeggios
of frames with the stochastic blurring technique are transformed in real time into chords with a
described previously (Figure 8). modified freeze tool. As shown in Figure 17, the
My implementation features a continuous spec- currently synthesized spectrum is added to the
tral recording of the input sound into a circular incoming one via a feedback loop. The “build
buffer of eight frames (an eight-column matrix). harmony” block uses the current output synthesis
This matrix is copied into the playback matrix upon data and the new window analysis to calculate a
receipt of the freeze message. matrix reflecting the addition of the new note. The
first solution for this computation is to average
the energy in each frequency bin over all frozen
Automatic Cross-Fade
spectra:
n
The stochastic freeze presented in Figure 16 switches ai
abruptly from one frozen sound to the next. A an = i = 1 (11)
n

100 Computer Music Journal

Figure 17. Real-time freeze
design with extensions.

where n is the number of frames to average, an is the a paradigm for extensions of the phase vocoder in
average amplitude, and ai is the amplitude in frame i. which advanced graphical processing is possible
The same formula can be used for phase differences. in performance time. The “composition of sound
The implementation requires a recursive equivalent treatments” described by Hans Tutschku (Nez
to Equation 11: 2003) is now available not only in the studio,
n × an + ai + 1 but also during a concert. I have also described an
an + 1 = (12) improvement to the real-time effect known as freeze.
n+ 1 I hope that these musical tools will help create not
Note that in the case of stochastic blurred resyn- only new sounds, but also new compositional
thesis over eight frames, this operation is done approaches. Some patches described in this article,
independently for each frame. complementary patches, and sound examples are
However, the solution I ended up using in concert available online at www.jeanfrancoischarles.com.
is given by
n
ai Acknowledgments
an = i√= 1 (13)
n
Thanks to Jacopo Baboni Schilingi and Hans
which is written and implemented recursively as Tutschku for their invitation to present this material
√ within the Prisma group; the exchanges with the
n × an + ai + 1
an + 1 = √ (14) group have been the source of important develop-
n+ 1 ments. Thanks to Joshua Fineberg for helping me
Indeed, the latter formula produces a more powerful through the writing of the article. Thanks to Örjan
sound, compensating for the low-amplitude frames Sandred for his useful comments after his attentive
that may be recorded. reading of my draft. Thanks to Andrew Bentley for
his invitation to teach this material.

Conclusion References

This article provides an overview of techniques to Boersma, P., and D. Weenink. 2006. “Praat: Doing Phonet-
perform spectral audio treatments using matrices in ics by Computer.” Available online at www.praat.org.
the environment Max/MSP/Jitter. I have presented Accessed 11 October 2006.

Charles 101
Bogaards, N., A. Röbel, and X. Rodet. 2004. “Sound Anal- Röbel, A. 2003. “Transient Detection and Preservation
ysis and Processing with Audiosculpt 2.” Proceedings in the Phase Vocoder.” Proceedings of the 2003 Inter-
of the 2004 International Computer Music Conference. national Computer Music Conference. San Francisco,
San Francisco, California: International Computer California: International Computer Music Association,
Music Association, pp. 462–465. pp. 247–250.
Dolson, M. 1986. “The Phase Vocoder: A Tutorial.” Schnell, N., et al. 2005. “FTM—Complex Data Structures
Computer Music Journal 10(4):14–27. for Max.” Proceedings of the 2005 International
Jones, R., and B. Nevile. 2005. “Creating Visual Music in Computer Music Conference. San Francisco, California:
Jitter: Approaches and Techniques.” Computer Music International Computer Music Association, pp. 9–12.
Journal 29(4):55–70. Schnell, N., and D. Schwarz. 2005. “Gabor: Multi-
Klingbeil, M. 2005. “Software for Spectral Analysis, Representation Real-Time Analysis/Synthesis.” Pro-
Editing, and Synthesis.” Proceedings of the 2005 Inter- ceedings of the 8th International Conference on
national Computer Music Conference. San Francisco, Digital Audio Effects (DAFx-05). Madrid: Universidad
California: International Computer Music Association, Politécnica de Madrid, pp. 122–126.
pp. 107–110. Schwarz, D. 2006. “Concatenative Sound Synthesis: The
Kreichi, S. 1995. “The ANS Synthesizer: Composing on Early Years.” Journal of New Music Research 35(1):3–22.
a Photoelectronic Instrument.” Leonardo 28(1):59– Sedes, A., B. Courribet, and J. B. Thiebaut. 2004. “From the
62. Visualization of Sound to Real-Time Sonification: Dif-
Laroche, J., and M. Dolson. 1999. “Improved Phase Vocoder ferent Prototypes in the Max/MSP/Jitter Environment.”
Time-Scale Modification of Audio.” IEEE Transactions Proceedings of the 2004 International Computer Music
on Speech and Audio Processing 7(3):323–332. Conference. San Francisco, California: International
Lewis, T. P. 1991. “Free Music.” In T. P. Lewis, ed. A Source Computer Music Association, pp. 371–374.
Guide to the Music of Percy Grainger. White Plains, Todoroff, T., E. Daubresse, and J. Fineberg. 1995. “Iana∼
New York: Pro-Am Music Resources, pp. 153–162. (A Real-Time Environment for Analysis and Extraction
Lyon, E. 2003. “Spectral Reanimation.” Proceedings of the of Frequency Components of Complex Orchestral
Ninth Biennial Symposium on Arts and Technology. Sounds and Its Application within a Musical Context).”
New London, Connecticut: Connecticut College, pp. Proceedings of the 1995 International Computer Music
103–105. Conference. San Francisco, California: International
Marino, G., M.-H. Serra, and J.-M. Raczinski. 1993. “The Computer Music Association, pp. 292–293.
UPIC System: Origins and Innovations.” Perspectives Wenger, E., and E. Spiegel. 2005. MetaSynth 4.0 User
of New Music 31(1):258–269. Guide and Reference. San Francisco, California: U&I
Moore, F. R. 1990. Elements of Computer Music. Engle- Software.
wood Cliffs, New Jersey: Prentice-Hall. Young, M., and S. Lexer. 2003. “FFT Analysis as
Nez, K. 2003. “An Interview with Hans Tutschku.” a Creative Tool in Live Performance.” Proceed-
Computer Music Journal 27(4):14–26. ings of the 6th International Conference on Dig-
Polansky, L., and T. Erbe. 1996. “Spectral Mutation in ital Audio Effects (DAFX-03). London: Queen
SoundHack.” Computer Music Journal 20(1):92–101. Mary, University of London. Available online at
Roads, C. 1995. The Computer Music Tutorial. Cambridge, www.elec.qmul.ac.uk/dafx03/proceedings/pdfs/dafx29.
Massachusetts: MIT Press. pdf.

102 Computer Music Journal