How Software Changes a Song's Tempo Without Changing Its Pitch

Slow a record down and it drops in pitch. Speed it up and everyone sounds like a cartoon. For most of recorded history, speed and pitch were a single control.

Software can separate them. It can slow a song by 8 percent and leave every note where it was, or move the key up two semitones and leave the speed alone. This is the engine room of adaptive music, since anything that must follow a runner's steps or a singer's range depends on it. Here is how it works, in plain words.

Why speed and pitch travel together

A recording is a long list of measurements of air pressure. Play the list back faster and every wave repeats more often, so the pitch rises by the same ratio. Slow it down and the pitch falls. A 2016 review of time-scale modification in the journal Applied Sciences, by Jonathan Driedger and Meinard Müller, notes that you get the same effect by changing the speed of a record player.

The numbers are not small. A semitone is a frequency ratio of about 1.059, so raising a song two semitones by speeding it up alone would also make it about 12 percent faster. Doing it properly means changing one without the other.

The basic trick: slice, then re-space

To change speed alone, most methods follow the same plan. The review describes cutting the sound into short frames, usually 50 to 100 milliseconds long, each of which holds a snapshot of the local pitch. The frames are then laid out on a new timeline, spread apart to slow the music down or bunched together to speed it up, and the overlaps are blended.

Because each slice keeps its own pitch, the pitch stays put. The trouble is at the joins.

Problem one: waves that no longer line up

Imagine copying a steady hum in short pieces and pasting them back with different gaps. Unless each piece starts at just the right point in the wave, the joins clash and the hum warbles. The review says the simplest method, plain overlap-add, gives harmonic sounds such as singing voice, piano, violin and guitar "a characteristic warbling sound."

A smarter method, waveform similarity overlap-add (WSOLA), was described by Verhelst and Roelands in 1993. It lets each frame slide a little, up to about 25 milliseconds, to the spot where its wave best matches what came before. That works well for a single voice or instrument. With a full mix, it keeps only the most prominent repeating pattern, so the rest can still warble.

Problem two: the drums

Drum hits cause the opposite trouble. A snare crack is not a repeating wave. It is a single event, and exactly when it lands is part of the rhythm. The review's example is a violin playing with castanets: to keep the violin, you must keep its pitch and tone, and to keep the castanets, you must keep each click crisp and in its place.

WSOLA can play a hit twice in quick succession when it stretches a sound, a fault called transient doubling, or drop a hit when it speeds one up. The phase vocoder, a frequency-based method introduced in the 1960s by Flanagan and Golden, keeps every pitch smooth but spreads each hit out. Listeners describe the result as reverberant, with "less bite" or a "loss of presence," the review says.

The modern answer: split, then treat each part its own way

Since no single method suits every sound, newer approaches combine them. One, published by Driedger, Müller and Ewert in 2014, first splits the music into a harmonic part and a percussive part. On a spectrogram, harmonic sounds draw long horizontal lines and percussive sounds draw thin vertical ones, so a simple filter can sort them. The harmonic part goes through the phase vocoder. The percussive part goes through plain overlap-add with very short frames of about 10 milliseconds. Then the two are added back together.

The percussive part still has its hits doubled, but the copies land so close together that the review says the doubling is not noticeable.

Here is the map of trade-offs at a glance:

• Plain overlap-add: simple and fast. Good for drums and clicks, but it makes tones and voices warble.
• WSOLA: lines up repeating waves. Good for one voice or instrument, but it can double or skip drum hits and struggles with full mixes.
• Phase vocoder: keeps every pitch smooth, even in a full mix, but it smears drum hits and can sound reverberant.
• Split and combine: treats harmonic and percussive sounds separately, at the price of extra steps.

Changing key is the same problem in reverse

To move a song up or down in pitch without changing its length, software does the speed trick twice. It first resamples, which changes pitch and length together, then time-stretches the result back to the original length. Raising a song one octave this way first makes it twice as high and half as long, and the stretch step doubles its length again.

Voices add a wrinkle. Every voice has resonant peaks, called formants, that give it its tone. Resample a singer upward and those peaks move too, so the singer takes on the cartoon sound known as the chipmunk effect. Better tools estimate the voice's overall spectral shape and restore it, but the review calls that estimation "highly complex and error-prone."

How far can you push it?

No method is perfect. The review states that "there is no single TSM method that can cope with all kinds of audio signals equally well." In practice, small changes are close to free and large ones show.

The team behind D-Jogger, the adaptive running player we describe in our piece on fitness music, used a commercial time-stretching engine. They judged tempo changes of up to 10 percent in either direction "acceptable in terms of sound quality." When they moved the idea onto a 2011 Android phone with no phase vocoder available, they prepared copies of each song in advance at plus or minus 1, 2 and 3 percent instead.

Browsers now ship a version of the same idea. MDN's documentation says the pitch of audio is "corrected by default" when you change a media element's playback rate, and that the correction can be switched off.

What it means for adaptive music

For music that must follow a listener, the lesson is to decide what to do live and what to prepare. A few percent of stretch is nearly invisible. Big moves, such as turning a ballad into a dance track, call for a different arrangement, not a faster copy. That is why designing a song so it can adapt starts with the writing and the arrangement, not with software.

Lining up two recordings adds another wrinkle. Real performances drift in tempo, so one fixed stretch factor rarely puts their beats together. The review describes stretching by a changing amount, steered by anchor points placed at matching beats, a technique used in scenarios such as automated DJing and soundtrack generation. Versions built on one shared tempo map need much less of this.

On Starchild, listeners can change a song's key and tempo live while it plays, and every song can be heard in different styles and voices before licensing. For small adjustments, a stretch is enough; for a change of feel, a different style is the better tool. Our explainer on pitch, tempo and vocal controls shows how those controls work from the listener's side.

Sources

• Applied Sciences: A Review of Time-Scale Modification of Music Signals (Driedger and Müller, 2016), PDF copy
• CCRMA, Stanford: Phase Vocoder, from Spectral Audio Signal Processing (Julius O. Smith III)
• PLOS ONE via PubMed Central: Encouraging Spontaneous Synchronisation with D-Jogger, an Adaptive Music Player That Aligns Movement and Music (Moens et al., 2014)
• MDN Web Docs: HTMLMediaElement playbackRate
• MDN Web Docs: HTMLMediaElement preservesPitch