Percussion Synthesis

My most recent research involves synthesizing percussive sounds. I am using modal synthesis and feedback frequency modulation (FM) techniques to synthesize sounds that are reminiscent of nonlinear percussion instruments. The technical portion of this project is described in a paper that I presented at the 2019 Sound and Music Computing (SMC) conference and can be found here. Accompanying sound examples can be found here. My PhD dissertation, titled "Physically-informed Percussion Synthesis with Nonlinearities for Real-time Applications", along with accompanying materials can be accessed here.

In learning about previous percussion synthesis methods, I implemented percussion models using the 2D waveguide mesh and finite difference approximations. The sounds that I synthesized through my experimentation are included here. Also, the gif above shows a 2D waveguide mesh in action after being excited by a 2D raised sinusoid!

The two following examples are 2D mesh waveguides with 21x21 junctions, excited by an impulse, with time-varying allpss filters at the boundaries and convolved with (a) a noiseburst and (b) a room impulse response.

To illustrate the impact of size on the mesh waveguide synthesis, the next two examples follow the same conditions as above except the number of junctions has been changed to 51x51.

These next four examples are finite difference approximations for the von Karman plate model. The parameters are set according to the material properties of a steel plate. The finite difference scheme used in these examples is from Stefan Bilbao's paper "Sound Synthesis for Nonlinear Plates" (2005). In the first two examples, the plate is given linear conditions with an inital velocity of 20 m/s and 70 m/s, respectively. In the last two examples, the plate is given nonlinear conditions with 20 m/s and 70 m/s initial velocity. Notice that under linear conditions, the two examples sound quite similar except for amplitude differences. Under nonlinear conditions, more frequency components as well as a slight pitch glide appear in the 70 m/s initial velocity example.

Using a real-time, modal synthesis technique, I created two steel plate examples that have been convolved with a noise burst and a room impulse response.

Goodhertz: Megaverb

During the Summer of 2018, I had the honor of working with the wonderful people at Goodhertz on an 80's reverb emulation plugin, Megaverb. You can read more about Megaverb, listen to sound examples, and buy it here.

RBM Drum Sequencer

The RBM Drum Sequencer is implemented as a Pd patch and is capable of generating variations of an input beat pattern. The pattern variation uses a restricted Boltzmann machine (RBM) trained on Drum and Bass rhythms (or Drum and Bass + Brazillian-style beats). This system can generate rhythmic variations for patterns that have 3 instruments and intensity of each note is included in the algorithm.

Here is an example of the audio output. You will first hear 4 bars of the basic, input beat. The following bars are variations created with the RBM trained on a Drum n' Bass dataset as well as the repeat module in the drum sequencer. Feel free to download the Pd extern, patch, and Python RBM training code below to explore this tool in real-time.

Genetic algorithm drum machine

This project is a drum machine that is able to generate rhythmic variations of an input beat sequence. The drum machine is implemented as a Pure Data (Pd) patch. The rhythmic pattern variation is based on a canonical genetic algorithm that is implemented as a Pd extern object in c. While the generated rhythmic variations are interesting and seem to fit the input beat sequence, incorporating different output levels for the generated percussion beats would greatly improve the "natural-ness" of the generated variations.

Here is an example of the audio output. You will first hear one bar of the basic, input beat. The following bars are variations created with the genetic algorithm as well as other modules in the drum machine (mutes, circular shifts, and individual sample levels). Feel free to download the Pd extern and patch below to explore this tool in real-time.

Ghost Note VST

The Ghost Note VST is a MIDI Effect plugin (written with JUCE and C++) that creates variations in percussive rhythmic patterns. The user is able generate ghost notes at the 2nd, 3rd, and 4th 16th notes of a quarter note within a beat pattern by specifying probabilities for the appearance of those notes as well as the volume of those notes. The beats that appear are a repetition of the note that falls on the quarter note beat (on the 1st 16th note of the quarter note). The user is also able to remove notes from the original pattern by specifying the percentage of notes that should be removed.

Here are some examples of the audio output. You will first hear one bar of the basic, input beat. The following bars are variations created with the Ghost Note VST. The last example is my favorite, so feel free to skip the first two if you're short on time!

Downbeat estimation

During the summer of 2015, I had an amazing experience as a reseach intern in the Media Technology Lab at Gracenote. I developed a method for estimating the time of the downbeat in music. When you count the beats in 4/4 music as "1 2 3 4 1 2 3 4 ..." or 3/4 music as "1 2 3 1 2 3 ...", the downbeat is the "1". My task was to find the time at which the "1" occurs in different audio signals. I used a machine learning approach instead of a strictly DSP approach for this task. I trained my models only on the Ballroom Dataset and tested them on other songs that were not in the dataset. This work could not have been done without the help of my supervisor at Gracenote, Greg Tronel.

Here are some examples of the audio output. You will hear the test audio with a metronome. The bleep with the high frequency is the estimated downbeat. I am only posting good examples here, so if you would like to hear some bad examples, feel free to contact me:

Music pattern discovery with a Variable Markov Oracle

This is a collaborative project between Cheng-i Wang, Shlomo Dubnov, and myself. Using a Factor Oracle-like data structure, the Variable Markov Oracle, we search for repeated patterns in polyphonic music. This method is capable of discovering patterns in both audio and symbolic representations.

Physical model of the voice

This project is for integrating a physical model of the voice with a live saxophone player. I implemented a physical model of the vocal tract and vocal folds that can morph between vowel sounds. I also developed a technique to specify the sounding frequency of the voice synthesis. Frequency trajectories extracted from the acoustic saxophone signal are used to control the pitch of the voice. Future work involves mapping other sonic aspects of the saxophone sound to voice model control parameters.

MBDelay

This is a multiband delay VST plugin. MBDelay allows you to split a sound into three frequency bands. The amplitude, feedback, and delay time for each band can be controlled separately. I am working on getting this to work on Mac OSX 10.10, but I have been experiencing issues with Apple vecLib FFT functions since updating my OS. If you have any suggestions, please send them my way.

GrannyGrains

This is a granular synthesis VST plugin. With GrannyGrains, you can chunk up your audio into grains (large or small) and play them either sequentially or in a shuffled manner. You can control the binaural spread of the grains, the feedback, and percentage of reversed grains. There is also an option to quantize the timing of the grains to the beat of your DAW.

Here's an example of the audio output. The first few seconds are of a clean music box signal. I manipulate parameters on GrannyGrains for the rest of the sound file.