Spectro-temporal modulation models

Etienne Thoret Perception Representations Images Sound Music

Laboratoire d’Informatique & Systèmes
Institute of Language Communication & the Brain

Computational Audition Meeting – 18_Dec_2019

What is a sound?
Analysis of sounds by sound synthesis
Analysis of sounds by sound synthesis
Analysis of sounds by sound synthesis

What is a “brassy” sound?

Risset, J. C., & Mathews, M. V. (1969).

Analysis of musical-instrument tones. Physics Today, 22(2), 23–30.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Changing the object => modes repartition

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Changing the object => modes repartition

Changing the propagation => reverberation

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Changing the object => modes repartition

Changing the propagation => reverberation

Changing the way the object is interacting with another object or with someone

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Changing the object => modes repartition

Changing the propagation => reverberation

Changing the way the object is interacting with another object or with someone

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Changing the object => modes repartition

Changing the propagation => reverberation

Changing the way the object is interacting with another object or with someone

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Changing the object => modes repartition

Changing the propagation => reverberation

Changing the way the object is interacting with another object or with someone

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
Superposition of harmonics to form “an auditory object”

Proof of concept: the case of impact sounds

Changing the material => damping / roughness

Changing the object => modes repartition

Changing the propagation => reverberation

Changing the way the object is interacting with another object or with someone

Aramaki, M., & Kronland-Martinet, R. (2006).

Analysis-synthesis of impact sounds by real-time dynamic filtering.
IEEE TASLP, 14(2), 695-705.
 Analysis of sounds by sound synthesis
The spectro-temporal modulation spectrum:
Spectro-Temporal Receptive Fields (STRF)
 Analysis of sounds by sound synthesis
The spectro-temporal modulation spectrum:
Spectro-Temporal Receptive Fields (STRF)
 Analysis of sounds by sound synthesis
The spectro-temporal modulation spectrum:
Spectro-Temporal Receptive Fields (STRF)
 Analysis of sounds by sound synthesis
The spectro-temporal modulation spectrum:
Spectro-Temporal Receptive Fields (STRF)
 Analysis of sounds by sound synthesis
The spectro-temporal modulation spectrum:
Spectro-Temporal Receptive Fields (STRF)
 Analysis of sounds by sound synthesis
The spectro-temporal modulation spectrum:
Spectro-Temporal Receptive Fields (STRF)
 Analysis of sounds by sound synthesis
The spectro-temporal modulation spectrum:
Spectro-Temporal Receptive Fields (STRF)

4D representation:
Spectral modulations
Temporal modulations
Time
Frequency
 Analysis of sounds by sound synthesis
Analysis: redefining musical instruments timbre

Thoret, Caramiaux, Depalle, McAdams (Under review)

Cortical modeling of context effects in perceived differences among complex sounds
 Analysis of sounds by sound synthesis
Analysis: redefining musical instruments timbre

Scale / Rate Frequency / Rate Frequency / Scale

Generic dimensions Context-driven dimensions

Thoret, Caramiaux, Depalle, McAdams (2021) Nature Human Behaviour

 Analysis of sounds by sound synthesis
One other model: McDermott & Simoncelli (2011) “summary statistics” for textures

ns
ti o
ul a
od
lm
ra
po
m
Te

Correlations between subbands

 Analysis of sounds by sound synthesis
One other model: McDermott & Simoncelli (2011) “summary statistics” for textures

1. Imposing marginal cochlear

and modulation statistics
ns
ti o

2. Imposing correlations
ul a
od

between subbands
lm

(cochlear & modulations)

ra
po
m

What’s cool?
Te

=> synthesizing “equivalent” sounds

Correlations between subbands

 Analysis of sounds by sound synthesis
One other model: McDermott & Simoncelli (2011) “summary statistics” for textures

1. Imposing marginal cochlear

and modulation statistics
ns
ti o

2. Imposing correlations
ul a
od

between subbands
lm

(cochlear & modulations)

ra
po
m

What’s cool?
Te

=> synthesizing “equivalent” sounds

Correlations between subbands

 Analysis of sounds by sound synthesis
One other model: McDermott & Simoncelli (2011) “summary statistics” for textures

1. Imposing marginal cochlear

and modulation statistics
ns
ti o

2. Imposing correlations
ul a
od

between subbands
lm

(cochlear & modulations)

ra
po
m

What’s cool?
Te

=> synthesizing “equivalent” sounds

Doesn’t always works!

Correlations between subbands

 Analysis of sounds by sound synthesis
One other model: McDermott & Simoncelli (2011) “summary statistics” for textures

1. Imposing marginal cochlear

and modulation statistics
ns
ti o

2. Imposing correlations
ul a
od

between subbands
lm

(cochlear & modulations)

ra
po
m

What’s cool?
Te

=> synthesizing “equivalent” sounds

Doesn’t always works!

Correlations between subbands

 Analysis of sounds by sound synthesis
One other model: scattering => “wavelet of wavelet of ... wavelet”

Anden & Mallat (2012) DAFx

 Analysis of sounds by sound synthesis
Still other models... Varnet et al. (2018)

Fourier vs Auditory

McWalter & Dau (McDermott 2.0)

“Theunissen vs Shamma”
 Analysis of sounds by sound synthesis
Issue: understanding the links between models

Summary statistics of amplitude modulation

filterbank

vs.

Spectro-temporal modulations

vs.

Scattering moments

(vs. Latent spaces in VAE synthesizing sounds)

 Merci

Former institutions

[email protected]