How Does Sound Adapt to the Room in Mixed Reality?

A sound that does not match the room sounds fake, even if you cannot say why. A violin in your living room should not ring like a cathedral. Researchers describe the failure as "a sensory mismatch between visual and auditory cues." Headsets avoid it by sensing the room, simulating how sound would travel through it, and applying the result to every virtual sound.

In mixed reality the real room stays audible, so the virtual sound has to fit into it. Apple says Vision Pro's speakers sit close to your ears and deliver Spatial Audio that "seamlessly blends with real-world sounds, keeping you aware of your surroundings." Here is how each step works, what developers can control, and what it means for music.

What a room does to a sound

Sound reaches your ears by a direct path and by bouncing off walls, floors and furniture. Surfaces matter. In Apple's WWDC26 session on RealityKit, its toolkit for spatial apps, the presenter says the geometry and materials of an environment "have a big impact on how an audio source sounds." The same source sounds very different in a small living room and a large museum. UploadVR reports that in Meta's audio tools, developers tag virtual objects with sound materials such as metal, brick, carpet, glass and wood, which change how they absorb and reflect sound.

Engineers track several effects. Early reflections are the first echoes to reach you, and reverb is the dense tail that follows. Meta's announcement, as UploadVR quotes it, explains that occlusion models how objects block sound, obstruction simulates how sound changes when it passes through different materials, and diffraction lets sound bend around obstacles.

Step one: the headset senses the room

Apple Vision Pro turns your surroundings into geometry that sound can bounce around. In the WWDC26 session, the presenter says that in a shared space the system uses its "room-sense reverb geometry," which is "a reverb mesh that the Apple Vision Pro has built based on your real-world surroundings."

Research goes further on paper. A UIST 2025 paper describes SAMOSA, an on-device system that fuses "real-time estimations of room geometry, surface materials, and semantic-driven acoustic context" to synthesize a room impulse response, a fingerprint of how a room answers a sound. The authors say existing methods "often struggle with real-time adaptation to diverse physical scenes."

Step two: it simulates how sound travels

Apple and Meta both describe ray-traced acoustics. As with light, the software follows rays from a source as they bounce around a space. Apple's session says RealityKit can simulate the reflections and reverb of an environment "using raytraced geometrical acoustics." It adds that accurate direction and timing of the direct and reflected paths "are required for a realistic spatial audio experience." Meta's Acoustic Ray Tracing "simulates how sound propagates from its source to the listener based on actual game geometry."

The idea is not new. UploadVR noted in 2024 that Valve's Steam Audio has had audio ray tracing since 2018. Meta claims its version is efficient enough to run on the mobile chips in Quest headsets. Meta also offers a lighter Shoebox model, where a developer sets the size of a rectangular room and its wall materials by hand.

Step three: the room is applied to every sound

Apple's documentation says that in visionOS, "an acoustic simulation of a person's real environment reverberates the spatial audio sources by default." Developers can set how much of each sound goes through that simulation and how much goes straight to the listener's ears. The simulation also has to keep up. Apple's session notes that as a person and the audio source move around, "the timing and orientation must be updated accordingly to maintain realism."

Real room, virtual room, or both

• Your room (mixed): in a shared space or a mixed-style immersive space, Apple says the system reverberates audio from its simulation of your real surroundings and ignores the app's own reverb setting.
• A virtual place (full immersion): developers can set a reverb so sound matches the visuals. Apple's example is that a desert, a living room and a concert hall each have unique acoustics, and when visuals and sound are "congruent, the environment becomes more effective as a whole."
• In between (progressive): the Digital Crown adjusts how the simulation of your real room and the app's reverb are blended.
• Meta's tools: the documentation describes a simulation built from the game's own geometry, prepared ahead of time. According to UploadVR, developers tag objects with sound materials, then bake the scene's geometry.

What it means for music

Music in a headset can sit inside your head like headphones, or live in the room like a speaker or a band. Our piece on spatial audio and adaptive scoring covers that choice. Once music lives in the room, the room shapes it. The WWDC26 session shows a sample in which "you can hear a virtual band play as you navigate around a large museum environment." In the same sample, "you can even control the sound coming from each instrument independently." Another Apple sample, Mixing spatial music, lets developers "preview ray-traced reverb by adjusting a spatialized multitrack audio mix in an immersive scene."

That suggests an idea for adaptive scores. If each layer of a layered score were its own source, the music could grow by adding instruments at different places in the room. Apple's museum sample controls instruments independently, but it is not described as adaptive, so treat this as an idea, not a documented feature.

Three habits follow. Keep instruments as separate parts so each can be placed. Go easy on reverb baked into the recording, because the system adds a room of its own and two rooms stacked on top of each other can sound strange. And listen in more than one room. These are judgment calls, not rules from the documentation, so check your platform's guidance.

A two-minute room test

• Clap once in a tiled bathroom, then once in a bedroom with a bed and curtains. Listen to how long the sound hangs.
• Hard, bare surfaces reflect more sound and soft ones absorb more. That difference is what the software calculates.
• Play the same track in each room and notice how much of the room you hear in the music.

How far can you trust it?

Apple describes what its system does but publishes no accuracy figures in the pages we read, and the SAMOSA authors say real-time adaptation is still hard. Treat a headset's promise to match your room as a goal that is partly met, and judge it by listening.

Sources

• Apple: Apple Vision Pro
• Apple Developer Documentation: ReverbComponent (RealityKit)
• Apple Developer: Explore advances in RealityKit (WWDC26 session 279)
• Apple Developer Documentation: Mixing spatial music (RealityKit)
• Meta Developers: Acoustic Ray Tracing overview (Unity)
• Meta Developers: Meta XR Audio SDK room acoustics (Unity)
• UploadVR: Quest's Audio SDK Now Has Acoustic Ray Tracing Simulation (2024)
• arXiv: Enhancing XR Auditory Realism via Multimodal Scene-Aware Acoustic Rendering (SAMOSA, UIST '25)