Why Does a Rhythm Game Say You're Late When You Hit the Beat?

You tap on the beat, the game says you were late, and it is not being unfair. It compares your tap with the moment it scheduled a note to sound, and the sound reaches you after that moment. The game and your ears run on different clocks. The gap is easy to create by accident, and gift season makes it likelier: a new TV or a pair of wireless earbuds can add delay the game knows nothing about.

It is not a flaw in you. It is a timing puzzle that every interactive music system has to solve, from games to music apps. Here is where the delay hides, how games keep time anyway, and what calibration really does.

What the game is checking

A chart is a list of notes with times, and the file formats show how plainly. A StepMania chart sets tempos as "Beat=BPM" pairs plus an offset, "the offset between the beginning of the song and the start of the note data in seconds." In an osu! beatmap, each timing point gives a start time "in milliseconds from the beginning of the beatmap's audio" and the length of a beat.

When you tap, the game measures how far you were from the note's time. In osu!, "each hit object has a time frame called a hit window in which players are required to hit them." The window is "centred on the hit object's correct time," and its size follows a setting called Overall Difficulty, or OD. The top judgment is 80 minus 6 times OD milliseconds, which works out to 50 ms on either side at OD 5 and 20 ms at OD 10. In osu!mania the top window is about 16 ms on either side.

For scale, Android's audio documentation says its requirement for round-trip latency is 20 ms or lower, "even though musicians generally require 10 ms." A delay of a few hundred milliseconds would put every tap far outside these windows.

Where does the delay come from?

Godot's documentation lists the usual suspects. Audio is mixed in chunks, not continuously, and mixed chunks are not played immediately. "Graphics APIs display two or three frames late." Its list ends with a warning: "When playing on TVs, some delay may be added due to image processing." On a screen that draws 60 frames a second, one frame lasts about 17 ms, so two or three frames add roughly 33 to 50 ms before the TV's own processing.

Then there is the output device. MDN describes the Web Audio property outputLatency as an estimate of the time between the browser handing audio to the system and "the first sample in the buffer" being processed by the output device, and says it "varies depending on the platform and the available hardware." Wireless headphones add their own delay. SoundGuys ran 100 tests per codec on each of four Android phones and found that even the two best averaged 244 ms and 250 ms over Bluetooth, while the slowest averaged 484 ms. The author wrote that delay "above 150ms becomes quite noticeable." A March 2021 update says things have improved. Google's own data put popular phones at "a hair under 40ms," a figure the author asked readers to take "with the usual helping of salt until we can confirm."

How games keep time anyway

The fix is to stop trusting the wrong clock. In "A tale of two clocks," Chris Wilson explains that web timers "can easily be skewed by tens of milliseconds or more" by layout, rendering and garbage collection, while the audio system's hardware clock stays steady. His example checks every 25 ms and schedules notes 100 ms ahead on the audio clock.

Game engines say the same. Unity describes its audio time as "much more precise" than game time because it is "based on the actual number of samples the audio system processes." Godot's guide to syncing gameplay with audio offers two methods. One counts time on the system clock and subtracts the startup delay. It suits a song of a few minutes, but "the timing information will slowly drift away." The other reads the audio hardware's playback position, adds the time since the last mix and subtracts the output latency.

What does calibration actually do?

Calibration measures the delay you actually have, then removes it. Say your wireless earbuds delay the sound by 180 ms, a made-up figure. You tap when you hear each beat, so the game sees every tap 180 ms late. A calibration notices the steady lag, stores 180 ms as an offset and judges each tap as if it had come 180 ms earlier. In osu! the setting is called universal offset: "The offset (in milliseconds) that all beatmaps will use (in addition to the local offset)." A button in the same menu opens an offset wizard.

A steady delay can be subtracted. A wobbling one cannot. Godot's guide warns that the audio position "may be a bit jittery" because of how threads work, and suggests discarding any reading that is lower than the previous frame's. Calibration also needs repeating whenever the path changes, because new headphones, a new TV or a different input mode each add a new delay.

What to try tonight

• Run the game's calibration again whenever you change headphones, speakers or TV.
• Use wired audio for music games when you can. Wireless links add delay, as noted above.
• If your TV has a game or low-latency mode, try it. It aims at the image-processing delay that Godot's guide mentions.
• If only the hardest songs feel unfair, check the difficulty setting. In osu!, at OD 10 the top judgment allows just 20 ms either side of a note.
• Judge a fix by several runs, not one.

What this means for adaptive music

Every interactive music system has the same triangle: what the listener did, what the system noticed, and what the listener hears. Games make the delays visible because they score you against the beat. Music that follows a runner's steps or a listener's tempo slider faces the same limits.

Designers have two answers. One is to cut the delay with smaller audio buffers, but Godot's guide warns that "when latency is too small, sound mixing will require considerably more CPU," which raises the risk of skipping. The other is to hide the delay by letting music wait for the next boundary. Guy Whitmore made that case in 2003: "transition boundaries and timing are so crucial." Wilson describes the same trade in scheduling. A long lookahead is more reliable, but "tempo changes, etc., will take a tenth of a second to take effect."

That is one reason a song built for adaptation keeps all its versions on one timeline, so a switch can land on the beat without waiting. See how video games make music adapt and what game audio middleware taught music makers. For how tempo gets onto a chart in the first place, read why a BPM finder says half or double.

Sources

• Godot Engine documentation: Sync the gameplay with audio and music
• Unity documentation: AudioSettings.dspTime
• web.dev: A tale of two clocks (Chris Wilson)
• MDN Web Docs: AudioContext.outputLatency
• Android Developers: Audio latency
• osu! wiki: Hit window
• osu! wiki: Overall difficulty
• osu! wiki: osu! file format
• osu! wiki: Options
• StepMania wiki: The .sm file format
• Game Developer: Design With Music In Mind, A Guide to Adaptive Audio for Game Designers (Guy Whitmore, 2003)
• SoundGuys: Android's Bluetooth latency needs a serious overhaul (Robert Triggs, 2021)