How Video Games Make Music Adapt

A film composer knows exactly how long a scene lasts. A game composer does not. Red Dead Redemption composers Bill Elm and Woody Jackson described the problem in an interview with Game Developer: someone might be riding their horse through a certain territory for half an hour. The music has to keep making sense for as long as the player stays, and it has to change when the player does something new.

Game audio has spent more than three decades building techniques for that. Three of them do most of the work, and they are worth understanding even if you never make a game, because the same ideas now shape adaptive music everywhere else.

Where it started: iMUSE

In the early 1990s, LucasArts composers Michael Land and Peter McConnell built iMUSE, an interactive music system first used in Monkey Island 2: LeChuck's Revenge. LucasArts patented it (US Patent 5,315,057). An often-cited example comes from the town of Woodtick: walking into different buildings brings in variations of the town theme played on different instruments, and walking back out triggers a closing flourish that leads back into the main theme.

The core idea, that music can wait, branch and resolve at the right moment, runs through everything that followed.

Horizontal re-sequencing: changing what plays next

Horizontal re-sequencing cuts a score into segments and decides, while the game runs, which segment comes next. Think of it as rearranging music along the timeline: an exploration section can loop, branch into a tension section, then hand off to combat.

The craft is in the handoffs. FMOD Studio's documentation describes transition markers and transition regions that make the playback position jump to a destination, and lets composers quantize those jumps so they only happen on chosen beats or bars. A relative offset setting lets the music jump at any point in a bar or phrase and arrive at the same relative position in the destination, which FMOD notes is frequently used for music. In Wwise, Music Playlist Containers sequence segments and Music Switch Containers swap music based on game states and switches, with transition rules deciding how each change happens.

Composer Winifred Phillips described a clear case in her GDC 2021 talk, summarized on Game Developer. In Sackboy: A Big Adventure, the level "Sink or Swing" uses a symphonic waltz in 3/4 time split into seven horizontal segments that track the player's progress through the level.

Vertical layering: changing what plays together

Vertical layering, also called vertical remixing or re-orchestration, keeps the timeline running and changes the mix. Phillips defines it as playing multiple independent audio files simultaneously in the game's audio engine, stacked on top of one another in perfect synchronization, so each layer can be manipulated independently during gameplay.

Because every layer shares the same tempo and length, the game can bring in drums when danger appears, strip back to strings when things calm down, or add a choir for variety, and the music never has to stop. Phillips used this approach in the Apple Arcade game Spyder, where layers enter as a mission progresses so a simple arrangement grows more complex as the player advances.

The tradeoff is that layers cannot change the harmony or form on their own. Vertical systems are good at intensity and color, and less good at telling the player that the story has moved somewhere new. That is one reason composers mix the two approaches.

Stingers: short phrases on top

A stinger is a brief musical phrase played over whatever is already playing, usually to mark an event such as a discovery, a victory or a sudden threat. Audiokinetic's Wwise-201 course teaches stingers alongside states and switches as music-specific features for making a score respond to gameplay. Karen Collins, in Game Sound, notes that a stinger can also help mask an abrupt change in game state that would otherwise make the music feel disjointed.

Sackboy shows a close relative of the stinger. In "Sink or Swing", a success melody plays as a separate layer whenever Sackboy collects Dreamer Orbs, and it merges harmonically with whichever horizontal segment happens to be playing. The reward sound is part of the score.

The groundwork that makes it work

None of these techniques works unless the music is written for it. Red Dead Redemption is a well-known case of planning ahead. Elm and Jackson told Game Developer they made 24 hours of music, including the Undead Nightmare expansion, all in A minor, and that a tempo of 130, or 65 at half time, suited much of the movement in the game. In a game the user controls, they explained, it helps to be able to drop one song over another at almost any moment.

Shared key and tempo, phrase lengths that line up, and parts that sound complete on their own are the hidden decisions behind most adaptive scores. Phillips makes a broader point in the same talk series: no single method can be considered the best way, and the game's context decides which one fits.

Most scores are hybrids

Real games rarely use one technique. Phillips's Sackboy example is horizontal and vertical at once, with segments tracking progress while a choir layer comes and goes. Chenzhong Hou's post on Audiokinetic's blog points out how messy the vocabulary gets, since vertical techniques are also called vertical remixing, layering or multi-track, and proposes sorting methods by whether they are driven by the user or the game and whether they change the mix or the sequence.

The same toolkit now reaches beyond games. Switching a song between production styles without losing your place is a close cousin of FMOD's relative-offset jump: every version has to share a timeline so the change can land mid-phrase. Starchild's playback engine does this for songs in the browser, including on phones, keeping every production in sync while tempo and key change live, and the engine is available to license for other products. For the production side of that story, see how adaptive songs are produced.

Sources

• Game Developer: Pure vertical layering for game music composers (From Spyder to Sackboy, GDC 2021)
• Game Developer: Hybrid horizontal-vertical structure for game music composers (From Spyder to Sackboy, GDC 2021)
• GDC Vault: From 'Spyder' to 'Sackboy: A Big Adventure' in Interactive Music
• FMOD Studio documentation: Authoring Events
• Audiokinetic: About Dynamic Music Design, Part 1: Design Classification
• Audiokinetic: Wwise-201 Interactive Music course
• Game Developer: Myths, Mavericks, and Music of Red Dead Redemption
• Wikipedia: iMUSE