Latent Fourier Transform

We compare the audible spectrum (the Fourier-domain representation of the audio waveform) and latent spectrum (Fourier-domain representation of the latent sequence). We play a song. In the first video (left), we progressively low-pass the song's waveform, as done in traditional audio equalization. In the second video (right), we progressively low-pass the latent sequence representing the song.

The audible spectrum ranges from 20–20,000 Hz (representing the limits of human hearing), while the latent spectrum has a lower range (0–43 Hz). The musical patterns that the latent spectrum captures occur on larger temporal scales than the oscillations in the audio waveform. The visualizations show which audible or latent frequencies are being retained from the input.

Progressively low-passing the latent spectrum has the effect of smoothing the musical patterns in the piece, first smoothing fine details like transients, then smoothing rapid arpeggios, then smoothing larger-scale features like the chord progression.

Just like a traditional audio equalizer, LatentFT requires intentionally (rather than randomly) selecting frequencies to boost. For instance, a song with a tempo of 120 beats-per-minute (bpm) might have interesting patterns at 2 Hz, 1 Hz, 4 Hz, and 8 Hz, but a song of a different tempo might require selecting different latent frequencies. Thus, we present demonstrative examples selected by the authors. We recommend listening with headphones or high-quality earbuds.

We show several conditional generation examples. Our goal is to generate variations that are diverse and musically interesting, while resembling the input music at the selected temporal scale (latent frequency). First, we play the input music clip:

We then zoom in on a rhythmic pattern in the music between 9–9.5 Hz in the latent spectrum, to capture rapid drum patterns occurring within that range.

Finally, we generate several variations, preserving the selected rhythmic pattern:

The variations evoke the essence of the selected rhythmic pattern, while remaining diverse and musically interesting. For instance, in the second example, the pattern materializes in the synthesizer and hi-hat pattern.

In this example, we select 0.5–1 Hz from some guitar arpeggios. Isolating 0–0.5 Hz retains the overall chord progression, which we can use to generate more variations.

We show several qualitative examples for blending. Our goal is to capture characteristics from two inputs, and combine them together. We play the first input, and then the latent frequencies we isolate:

Here, we blend patterns from the first input with very low-frequency characteristics (0–0.25 Hz) from the second input. Note again that in the blending, the synth pattern from input 2 has been restylized to mix seamlessly with input 2.

In this example, we combine the large-scale features of a Christmas song with the transient details (high frequencies) of a guitar recording. Higher latent frequencies will capture transients, onsets, and timbral details, while removing aspects like key and chord progression, as we can see in input 2's isolation. As a result, the guitar recording's transients/onsets are imparted onto the Christmas song's chord progression, creating a timbre-transfer effect where the guitar's strumming drives the Christmas song's chord progression.

We show a third example below. Again, we select a larger band of high frequencies from input 2 to capture sharp onsets/transients. We impart these onsets onto input 1's global characteristics, resulting in a timbre-transfer effect, where input 2's onsets 'drive' input 1.