Additive Sound Synthesis

Introduction

So we understand the Fourier Transform: we can express any waveform as a sum of sinusoids that are appropriately weighted and are at discrete multiples of a chosen “fundamental frequency”.

How do we use these ideas to synthesize sound?

Inspiration

TO BE ADDED (sic!)

What is Additive Synthesis?

First we need to get used to the idea of an oscillator.

An oscillator is a source: it generates waveforms that we perceive as sound. Let us play with a few oscillator types here:

https://musiclab.chromeexperiments.com/Oscillators/

Each of these waveforms, by the Fourier series, is the sum of an ( infinite) number of sine wave outputs.

In Fourier series, we normally use just sine wave oscillators, and use many of them to add up to obtain the wave form we need. Now let us hear from Mr Shiffman again:

The Math of Waveform Addition

In general, we can write a sum of sine/cos waves as:

\[ g(t) = \large{a_0 + \sum_{k=1}^{\infty} \Bigg[ a_k*sin(k* 2\pi *f *t) + b_k*cos(k* 2\pi *f *t) \Bigg]} \tag{1}\]

\(g(t)\) is the desired time-waveform, \(\omega = 2\pi\times f\) is the fundamental frequency in radians/sec, and the \(a_k\) and \(b_k\) are weights for the individual components that are to be designed.

How does this look like? Mr. Shiffman again:

Now let us see how we can design something using the Additive Method.

Design Principles for Additive Synthesis

How do we do this with intent? We will follow the development in Farnell and Risset and Mathews, (Risset and Mathews 1969) and Moorer.

1. The idea is to take an original sound, analyze that using the Fourier Series, and then use those coefficients to synthesize the sound with code.
2. The coefficients, or parameters, need to be manipulated and transformed with time, in order for the synthesized sound to have a “live” feel.
3. The number of such parameters and their control over time could pose a formidable data management challenge. This leads to the idea of data reduction in order to have a manageable number of these, and generate the sound in its essentials.
4. One essential part of this is to use envelopes around the amplitudes of several sine waves, what is called the ADSR method. This could also lead to several oscillators being turned on or off based on need.

NoteWhat is ADSR?

ADSR stands for “Attack Decay Sustain Release”. These related to the way a note of music varies over time in a typical piece of music.

4. So one needs to break down the sound into “principal components” that are harmonically related ( as with the Fourier series) and then fill in inharmonic tones using additional oscillators.
5. Adding these components allows us to synthesize authentic-sounding musical instruments, with the appropriate timbre.

NoteWhat is Timbre?

When different instruments play the same note, we always know that they sound different. It is impossible to mistake a violin for a flute or saxophone for a piano! So how does this work?

Each musical instrument has a characteristic set of frequencies (harmonics and in-harmonics) that it generates when a specific note is played. Think of this a an audio signature. These drop off in amplitude over frequency and are a function of the structure, resonances, and even playing technique ( e.g. bowing of a violin) of the intstrument.

Table 1: Basic-wave-table-sounds. Source: Andrew Olney https://olney.ai

Another way to view the multiple spectral components of a musical sound is using a spectrogram: https://musiclab.chromeexperiments.com/Spectrogram

Additive Synthesis with Code

Using p5.js

Using R

music <-
  Music() +
  Meter(4, 4) +
  Line(c("C5", "D5", "E5", "F5"))

show(music)

Wait, But Why?

• Sometimes we wish to reproduce a sound that we have encountered in the wild
• We may have managed to capture a recording, or have access to a similar-sounding sound
• We wish to reproduce this sound in sync with other events, to create an experience for the viewer / user
• The Fourier Series expansion helps us with reliably breaking down the sound into components, which we can controllably synthesize in p5.js using the p5.sound library

References

1. James Moorer. (Nov 1976) The Synthesis of Complex Audio Spectra by Means of Discrete Summation Formulae. Journal of the Audio Society. PDF
2. Jean-Claude Risset, Max V. Matthews. (Feb 1969). Analysis of Musical Instrument Tones. Physics Today. https://sci-hub.se/https://doi.org/10.1063/1.3035399
3. https://www.cs.cmu.edu/~tcortina/15104-f20/lectures/24-MoreSound.pdf
4. R package gm: the grammar of Music. https://cran.r-project.org/web/packages/gm/vignettes/gm.html
5. Phil Burk,Larry Polansky, Douglas Repetto, Mary Roberts Dan Rockmore. Music and Computers: A Theoretical and Historical Approach https://musicandcomputersbook.com
6. Julius O. Smith. Physical Audio Signal Processing for Virtual Musical Instruments and Audio Effects. https://ccrma.stanford.edu/~jos/pasp/pasp.html
7. Tutorials on WebSoundArt. https://websoundart.org/blog
8. Andrew Olney.Computational Thinking through Modular Sound Synthesis https://olney.ai/ct-modular-book/index.html
9. Michael McCarthy. A Musician’s Compendium. https://musicianscompendium.netlify.app/

Resources

1. Allison Parrish. Synthesizing and Analyzing Sound. https://creative-coding.decontextualize.com/synthesizing-analyzing-sound/
2. Thomas John Martinez. A collection of p5.sound sketches that demonstrate the new p5 sound library. https://editor.p5js.org/thomasjohnmartinez/collections/Dp0zGclVL
3. p5.Sound Tutorial.https://pdm.lsupathways.org/6_resources/7_soundandmusic/p5.sound/
4. Sound in p5.js Playlist. https://www.youtube.com/playlist?list=PLRqwX-V7Uu6aFcVjlDAkkGIixw70s7jpW
5. Sounds with Tone.js. https://pdm.lsupathways.org/3_audio/
6. Mister Bomb. p5.Sound project tutorials https://www.youtube.com/playlist?list=PLIsdHp2z9wFl7A1wWb2VmQUUojEGsKELE
7. Jason Sigal. Visualizing Music in p5.js. https://therewasaguy.github.io/p5-music-viz/

Other Tools

1. Strudelhttps://strudel.cc
2. Pure Data
3. Envion https://www.peamarte.it/env/envion_v3.6.html

R Package Citations

Package	Version	Citation
gm	2.0.0	Mao (2024)

Mao, Renfei. 2024. gm: Create Music with Ease. https://doi.org/10.32614/CRAN.package.gm.

Risset, Jean-Claude, and Max V. Mathews. 1969. “Analysis of Musical-Instrument Tones.” Physics Today 22 (2): 23–30. https://doi.org/10.1063/1.3035399.