How close does the sync land? Sixteen real piano recordings, measured

Summary

Across the 14 recordings where attacks can be measured, the median synced note lands 7.9 ms from the attack we find in the recording, 81% land within one frame of a 30 fps video and 90% within two. Our way of finding attacks is itself off by a median of 4.9 ms when the truth is known (section 4), so part of that spread is the ruler, not the sync. In 3 of the recordings the pianist does not play the score as the file writes it (repeats taken or skipped); the sync finds the order they played it in by itself.

The earlier figures on our home page (a median under 5 ms, 43 ms at worst) come from a synthetic test, where a score is played back under a known tempo warp. This is the harder test: real pianos, real rooms, real rubato, and no answer key.

1 · The test set

Every recording here is in the public domain or CC0 (Wikimedia Commons, most of them from Musopen), and every score is an edition from the Mutopia Project, read as MIDI, the way someone would import it into SheetVideo: all in the public domain but the Nocturne's, which is CC BY-SA 3.0. We chose them for range: slow and fast, sparse and dense, a 1902 piano roll, a twelve-minute ballade, and three pieces whose recordings do not follow their score's repeats.

* The recording repeats or skips sections: the bar count is of the score as played. Tempo range: the 5th to 95th percentile of each bar's tempo against the tempo the file gives.

2 · How the sync works

2.1 What is compared

The recording is read at 22.05 kHz and cut into overlapping frames 23 ms apart (a 4096-sample window, so the bass is still resolved to the semitone). Each frame becomes two short vectors: a chroma vector, the energy in each of the twelve pitch classes, and an onset vector, which pitch classes just got louder, with one more channel for broadband hits. The score is not rendered to audio for this: the same two vectors are written straight from its notes, so nothing depends on how a synthesiser happens to sound.

2.2 The cost of pairing two moments

For score frame $i$ and recording frame $j$, with chroma vectors $x_i$, $y_j$, onset directions $o_i$, $p_j$ and onset strengths $a_i$, $b_j$:

with $\alpha =0.4$: the harmony counts, and so does whether a note is struck at that moment or only still ringing. Both vectors are unit length, so a perfect match costs 0 and nothing in common costs about 1.

2.3 The path, and what it may leave out

The sync is a monotone path through the matrix of these costs (figure 3), from the first note of the score to its last, found by dynamic programming. A recording holds more than the score: silence, a breath, someone talking before the first chord, a take that stops early. Rather than letting the path start and end anywhere for free, every frame of recording the path leaves unexplained is charged at one rate, $\kappa$, the cost a matched frame of this piece typically has. With $U$ the number of sounding frames of recording the path leaves out and $B$ the number of bent steps it takes, a path's total is

Nothing is normalised by the path's length, so a path that stops short of a slow ending does not look better for having covered less. A bent step (the score moving at two or three times the recording's pace, or the reverse) is charged $\beta \kappa$, so the path follows the tempo it has found rather than wandering to whichever chord matches best locally.

2.4 Coarse to fine, twice

A four-minute piece is a matrix of about 10⁸ cells, too many to search directly. The sync first fits a straight line (a global tempo and offset) at a coarse resolution, searches a band around it that allows 35% of cumulative rubato, and then refines only a narrow tube round the coarse path at the full 23 ms resolution. It then does this a second time with the score laid out at the tempo curve the first pass found, so a long ritardando becomes part of the layout and the step charges only ever fall on note-level timing.

2.5 When the pianist does not play the score as written

A MIDI file has no repeat signs, so a minuet or a waltz is written once and played with its repeats; a file that writes them out meets a pianist who skips some. Before the fine pass, the sync finds the order the bars are played in, with a coarse pass (about 4 frames per second) where the path may also jump at a bar line to the start of any other bar, back or ahead (after Fremerey, Müller and Clausen's JumpDTW):

where $s_a$ is the first frame of bar $a$, $e_b$ the last of bar $b$, and $J$ the price of a jump: three seconds' worth of unexplained recording. A repeat the score lacks is cheaper to jump back for than to leave unexplained; a bar that only resembles another is cheaper to walk through. The order is kept only if it saves more than one more jump's worth over playing the score straight through; otherwise the score is left as it is. Of the sixteen recordings, three came back with a different order (table 1), and they are the three that do not follow their file: both repeats in the Schumann, the first repeat only in Träumerei, and a 1902 piano roll of The Entertainer which, by the sync's reading, leaves out the last two strains and goes from the return of the first straight to the final bar. The editor says so when the sync is done (figure 4).

2.6 From the path to the notes

The path gives a time for every bar line and every note onset. Each onset is then moved onto an attack detected in the recording, if there is one within 80 ms, so the note lights on the attack you hear rather than on a smoothed estimate of it. Finally every bar is checked: how well its notes matched, whether the path stood still or skipped there, and whether its tempo leaps against its neighbours. Bars that fail are flagged in the editor for a person to look at, and can be fixed by dragging one note onto its attack.

3 · The tempo the pianists took

None of these recordings holds the tempo its file gives, and most do not hold any tempo. Figure 6 shows each pianist's tempo, bar by bar, against the written one; the sync is never told any of it. The widest, Chopin · Ballade No. 4 in F minor, moves between 0.62× and 1.69× the written tempo; the steadiest, Bach · Prelude in C major, BWV 846, stays between 0.92× and 1.11×.

4 · How we measured

A real recording comes with no answer key: nobody wrote down when each note was struck. So for each synced note we look for its attack in the recording itself, by following the energy at that note's own pitch (its fundamental and first two harmonics) through 2.9 ms steps, and taking the moment it rises fastest. The search stays within half the distance to the neighbouring notes, so it cannot land on another note's attack, and a note is only counted when its attack can be told apart:

• no other note starts within 120 ms of it;
• no note within 250 ms shares its partials (the same pitch, a fifth, an octave, an octave and a fifth, two octaves);
• its rise is at least 6 dB, and 4 dB stronger than any other rise in the window.

Chords count once, by their top note. That leaves 2,777 notes in 14 recordings. In the densest études almost no note has 120 ms to itself (Chopin · Étude Op. 10 No. 1 Waterfall, Chopin · Étude Op. 10 No. 5 Black Keys), so they are synced like any other piece but not scored here.

Checking the ruler. To see how good this detector is, we ran it where the truth is known: on every score of the set rendered by SheetVideo's own piano, starting each search from a guess up to 50 ms off. It finds the true attack with a median error of 4.9 ms; 77% of its answers are within 10 ms, 91% within 20 ms. The rendered piano's attack is cleaner than a real one's, so on the recordings the detector can only do worse, and some of the spread in section 5 is the detector's, not the sync's.

5 · Results

Table 2. Error is the distance from a synced note to its attack in the recording. Pieces in grey have too few notes with a measurable attack (fewer than 30) to score. Sync time is for the whole piece, in the browser, on our test machine (a Windows desktop).

6 · What goes wrong

• The tail. One note in ten lands further than 67 ms from where we find its attack. Some of that is the detector (a pedal swell or a sympathetic string can rise faster than the note itself); some is the sync taking a rolled chord or an ornament as one event. The bars where it matters most are the ones the sync flags.
• Very dense writing. In the études, notes arrive every 70 ms or less. The sync follows them, but at this resolution a note can be one sixteenth off within a bar without the harmony changing, and our measurement cannot see it either.
• Noise at the ends. Applause and talking at the start or end of a video can pull the first or last bars onto them. The editor detects applause-like noise and asks for a trim.
• Scores that differ from the playing. Repeats are found (section 2.5); a cut in the middle of a section, an added cadenza or an ossia are not, and show as flagged bars.

7 · Reproduce it

Everything here comes from the same code the editor runs, driven headless: the recordings and scores are listed with their sources in table 1, the measurement and the figures are made by scripts in our repository, and the numbers on this page are read from their output, not typed in.

Try it on your own recording How to sync a score to a video