Adaptive Music in VR, AR and Live Immersive Experiences

On a screen, music sits beside the picture. In a headset, a ride vehicle or a room you can walk through, music becomes part of the space around you, and that space keeps changing because you are moving through it. That is why immersive projects reach for adaptive music: a fixed track cannot know where you are looking, how fast you are moving or what you just touched.

Here are documented examples from virtual reality, mixed reality, location-based audio, theme parks and live performance, and what they have in common.

Virtual reality: your actions shape the score

Tetris Effect (2018), from Tetsuya Mizuguchi's team, is one of the clearest examples of music tied to play. Engadget's review described how, as blocks fall into place and you clear lines, the music syncs to your game, and how the music swells as you get further in. The game was built for PlayStation VR as well as a regular screen, and the reviewer noted that latency in VR is extremely low. Engadget summed up the result as a synesthesia of music, gameplay, visuals and haptic feedback through the controller, which is a useful way to think about immersive music in general: sound is one channel among several, and they all have to move together.

Low latency matters more in a headset than anywhere else. When sound and motion drift apart in VR, it feels wrong at a physical level, so adaptive music in VR has to respond in time with the player's movements, ideally on the beat.

Mixed reality: playing music with your hands

Sigur Rós worked with Magic Leap for years on Tónandi, a mixed reality piece whose name is a made-up Icelandic word meaning "sound spirit." Reykjavík Grapevine reported in December 2017 that users could interact with these sound spirits and change the sounds with their hands, that every user's experience would be different, and that the band composed music specifically for the project. Bassist Georg Hólm suggested it could be a new way to release an album.

Mixed reality adds a twist: the music is placed in your real room. A musical creature can hover near your table or drift toward a window, which turns listening into something closer to exploring.

Location-based audio: the city as the interface

Audio augmented reality does not need a headset. Bluebrain's album app The National Mall used GPS to change its music as listeners walked between hundreds of tagged zones in Washington, D.C. Engadget reported that the rhythm, melody, instrumentation and pace all shifted with location, with no user input at all.

The design lesson from location-based music is pacing. A walk takes as long as it takes, so the music has to loop, extend and transition gracefully, which is the same problem game composers solve with horizontal re-sequencing.

Theme parks: variety for repeat riders

Theme parks face a different version of the problem, since people ride the same attraction again and again. Guardians of the Galaxy: Cosmic Rewind at Epcot, which opened in May 2022, answers with variety. Theme Park Insider reported that each ride plays a random selection of one of six songs, following the approach of Guardians of the Galaxy: Mission Breakout at Disney California Adventure, and quoted Disney saying you won't know which track you'll hear until you blast off.

This is a light form of adaptivity, since the choice is random instead of responsive. It still shows how much a single musical decision changes an experience: the same ride feels different depending on which song you draw, which makes a good case for having more than one version of the music ready.

It also shows a practical constraint of physical rides. Every song has to fit the same track layout, the same launch and the same scenes, so each one needs to work against a fixed timeline. That is the same discipline that lets game music switch segments on the beat.

Live performance: the musician takes the game's seat

Adaptive music tools are also reaching the stage. In a post on Audiokinetic's blog, a sound designer describes building a live "Robotic Orchestra" with Wwise and MIDI controllers, binding tempo-synchronized music segments, stingers, transitions, states and game parameters to physical controls, including a capacitive sensor that responds to touch. As the author puts it, if those changes are normally triggered by the game, the musician is in the game's role now.

That inversion is useful for installations and concerts. The adaptive score is written once, and the room, the performer or the audience supplies the signals that steer it.

What immersive projects teach about adaptive music

The same requirements apply to songs. Starchild's real-time playback engine keeps every production of a song in sync and handles tempo and key changes live in the browser, including on phones, and it is available to license for other products. Immersive work tends to need exactly that combination of in-sync versions and live control. For the wider picture, read music that knows the room, and for the hologram side of live music, avatar concerts go on tour.

• Latency is part of the composition. In VR and live settings, music that responds late feels broken.
• Every version needs the same timeline. Switching between parts only works when they share tempo and bar lines, so each change lands on the beat.
• Plan for unknown duration. Walks, rides and play sessions vary in length, so the music needs loops and graceful exits.
• Variety rewards return visits. Even random selection, as at Cosmic Rewind, makes a familiar experience feel new.

Sources

• Engadget: Tetris Effect is therapy for distracted, anxious minds (2018)
• Reykjavík Grapevine: Sigur Rós team up with Magic Leap for mixed reality app (2017)
• Engadget: Location-aware album gives Washington DC dynamically modified soundtrack (2011)
• Theme Park Insider: Here are the songs on Guardians of the Galaxy: Cosmic Rewind (2022)
• Audiokinetic: Making Interactive Music in Real Life with Wwise