The mathematics, using nothing but arithmetic

What this is all for

Two people say the same word. Two people write the same letter. A sonar pings twice. In every case there are two things that happen over time, and the question is always the same:

How far apart are they?

Answer that well and you can recognise a word, tell two handwritings apart, or spot when a machine's sound has changed. The whole method is one measuring stick, built in six moves. Everything else in the papers is a variation on those six moves.

You need three things you already have: multiplying, dividing, and the square-root button. Later, one more button called arccos. That is the entire toolkit.

Move 1A moment becomes a list of fractions

Take one instant of sound — say ten thousandths of a second. Split it into buckets: how much low sound, how much middle, how much high. Suppose the meter reads:

low 50    middle 30    high 20        total = 100

Now divide each one by the total:

50 / 100 = 0.5
30 / 100 = 0.3
20 / 100 = 0.2

the list is:  0.5   0.3   0.2        and it adds to 1

That's it. That is the first move, and it is just division. Two rules hold no matter what: nothing is negative, and everything adds to exactly 1.

The buckets don't have to be sound. They can be colours in a light reading, distances in a sonar echo, or directions a pen moved. The arithmetic doesn't know the difference.

The name in the papers
A list like this is called a probability distribution, and one instant's worth is called a frame. When you see the word "distribution", read: a list of fractions that adds to 1.

One more name. If you have 3 buckets, all the possible lists together form a flat triangle sitting in space, with corners at "all low", "all middle", "all high". The papers call that triangle the simplex. It's just the word for "every list that could possibly come out of Move 1".

Move 2Every such list sits on a ball

Take the square root of each number in the list. That's one button, three times:

√0.5 = 0.707107
√0.3 = 0.547723
√0.2 = 0.447214

Now square them again and add them up:

0.707107 × 0.707107 = 0.5
0.547723 × 0.547723 = 0.3
0.447214 × 0.447214 = 0.2
                      ───
                      1.0

Of course it adds to 1 — squaring undoes the square root, so you're just adding the original numbers back up, and those always added to 1.

Nothing clever happened. Taking the square root didn't put the data on a ball; it revealed that every possible sound-moment was already living on one. The flat triangle from Move 1 became a curved patch on a globe.

And that matters because a globe is a shape we know how to measure on.

Move 3Distance on a ball is an angle

Two cities on Earth. You don't measure the distance by drilling straight through the planet — you measure the curved path across the surface. And that curved path is an angle: London to New York is about 51 degrees of the way around.

So: our two sound-moments are two points on a ball. The distance between them is the angle between them. Here's how to get it, with arithmetic.

Worked example — do this on a calculator

Two moments:

first list:   0.5   0.3   0.2
second list:  0.4   0.4   0.2

Step one. Multiply them together, bucket by bucket, and square-root each answer:

0.5 × 0.4 = 0.20      √0.20 = 0.447214
0.3 × 0.4 = 0.12      √0.12 = 0.346410
0.2 × 0.2 = 0.04      √0.04 = 0.200000

Step two. Add those three up:

0.447214 + 0.346410 + 0.200000 = 0.993624

Call that the closeness number. It runs from 0 to 1:

• 1 means the two lists are identical
• 0 means they have nothing in common at all
• 0.993624 means these two are very nearly the same

Step three. Turn that closeness into an angle. There's a button for this: arccos.

arccos(0.993624) = 6.4737 degrees

Step four. Double it. (Purely a convention — everyone in the field scales it this way, and it changes nothing about which things are closer than which.) In radians rather than degrees, which is what the code uses:

angle           = 0.112987
doubled         = 0.225974      ← the distance

Sanity check. Compare a list with itself: the closeness number comes out to exactly 1, and arccos(1) = 0. A thing is zero distance from itself. Good — the measuring stick isn't broken.

The names in the papers
The closeness number is the Bhattacharyya coefficient. Terrifying word, and it is multiply the pairs, square-root them, add them up. The final doubled angle is the Fisher–Rao distance. It is not something anyone invented for this project — it's the standard, correct notion of distance between two distributions, settled by a statistician named Rao in 1945. There is no tuning knob and nothing to fit.

Move 4Why not just subtract?

Fair question. Why go through a ball and an angle when you could just subtract the two lists and see how big the gaps are?

Because subtraction says a gap of 0.1 is a gap of 0.1 no matter where it happens. That is wrong, and here is the proof in two lines of arithmetic:

Look at what happened. By plain subtraction the second pair is closer (0.16 beats 0.20). By angle it is nearly twice as far apart.

And the angle is right. Going from 0.5 to 0.4 is a nudge — the bucket is still half full. Going from 0.10 down to 0.02 means that bucket almost went silent. Five-sixths of what was there is gone. That is an enormous change, and subtraction can't see it because the numbers involved are small.

Move 5Cheaper measuring sticks — and one that cheats

The angle is the exact, correct answer, but arccos is slow when you need to do it millions of times. So there are alternatives. All of them are plain arithmetic, all give the same ordering of near and far. On the two lists from Move 3:

0.225974 × 0.225974 = 0.051064
0.051064 ÷ 4        = 0.012766

the fast chi-squared said:  0.012698

The rule that separates a fair stick from a cheat

A fair measuring stick has to obey four common-sense rules:

1. Distance is never negative.
2. It's zero only between a thing and itself.
3. A to B is the same as B to A.
4. No shortcuts. Going straight from A to C can never be longer than going A to B to C. Detours don't save distance.

Rule 4 is the one that catches people out. Here are three ordinary lists, and here is chi-squared failing it:

A = 0.7 0.2 0.1     B = 0.3 0.4 0.3     C = 0.1 0.2 0.7

Fisher–Rao      A→B 0.834014   B→C 0.834014   detour 1.668027
                                              direct 1.507434     detour is longer ✓

chi-squared     A→B 0.163333   B→C 0.163333   detour 0.326667
                                              direct 0.450000     detour is SHORTER ✗

root chi-sq     A→B 0.404145   B→C 0.404145   detour 0.808290
                                              direct 0.670820     detour is longer ✓

The names in the papers
A stick obeying all four rules is a metric. One that obeys the first three but breaks rule 4 is a divergence. The frame-to-frame stick, whichever one you pick, is called the ground metric — "ground" because everything else is built on top of it.

Move 6A word is a curve

One instant of sound is one point on the ball. But a word isn't an instant. Chop a spoken word into two hundred instants and you get two hundred points, in order — a line traced across the surface of the ball.

A spoken word is a curve. A handwritten letter is a curve. A sonar echo is a curve. A machine's sound over an hour is a curve.

Move 7Lining up two curves that run at different speeds

Here's the problem. You say "seven" in four tenths of a second, someone else says it in six tenths. Comparing instant 1 to instant 1, instant 2 to instant 2 is nonsense — you're not at the same place in the word at the same moment.

The fix is from 1978 and it is not ours: let one instant match several instants of the other, so a slow speaker's drawl gets absorbed. Two rules only:

• Never go backwards. Both recordings run forwards, always.
• Stay near the diagonal. You can stretch a bit, not unrecognisably — otherwise you could warp any word into any other word.

Then, of all the legal ways to line them up, take the cheapest.

The simplest possible example

Two buckets only, and someone drawls the first sound:

you:   1 0     1 0     0 1        (three instants — you held the first sound)
me:    1 0             0 1        (two instants — I said it quickly)

Compare instant-to-instant and you have a problem: your instant 2 (1 0) lands against my instant 2 (0 1), which is a completely different sound. That comparison scores 1.5708 — it says these are two different words.

Now allow the stretch. Your first two instants both match my first instant. Your third matches my second.

your instant 1  →  my instant 1     cost 0
your instant 2  →  my instant 1     cost 0
your instant 3  →  my instant 2     cost 0
                                    ─────
                            total   0.0        over 3 steps  →  0.0

Exactly zero. Same word. Which it is.

A full one, small enough to check on paper

Curve P has 3 instants, curve Q has 4. Both slide from bucket 1 to bucket 2 — same journey — but Q dawdles at the start.

P:   0.9 0.1     0.5 0.5     0.1 0.9
Q:   0.9 0.1     0.8 0.2     0.5 0.5     0.1 0.9

Step one. Work out the cost of every possible pairing, using Move 3 on each pair. Twelve pairings, twelve angles:

                Q1        Q2        Q3        Q4
     P1      0.000000  0.283794  0.927295  1.854590
     P2      0.927295  0.643501  0.000000  0.927295
     P3      1.854590  1.570796  0.927295  0.000000

Step two. Fill in a second grid of running totals. Each cell is its own cost, plus the cheapest of the three cells you could have arrived from (the one to its left, the one above, or the one diagonally up-left). Start at the top-left and work across:

                Q1        Q2        Q3        Q4
     P1      0.000000  0.283794  1.211089  3.065680
     P2      0.927295  0.643501  0.283794  1.211089
     P3      2.781886  2.214297  1.211089  0.283794   ← the answer lands here

Step three. Walk backwards from the bottom-right corner, each time stepping to whichever of the three neighbours was cheapest. That traces out the winning line-up:

P1 → Q1      P1 → Q2      P2 → Q3      P3 → Q4

steps taken   = 4
total cost    = 0.283794
0.283794 ÷ 4  = 0.070949      ← the distance between the two curves

Notice what the stretch did: P1 got matched to both Q1 and Q2. That's Q's dawdle being absorbed.

And compare against not stretching at all — line them up 1-to-1 and cut Q short:

no stretch:  0.523599          with stretch:  0.070949        7.4 times better

The stretch found that these are nearly the same journey. The naive comparison called them wildly different.

The name in the papers
Dynamic Time Warping, usually just DTW. Sakoe and Chiba, 1978. Standard, borrowed, not ours — and we say so.

Move 8The master distance

It has exactly two parts, and keeping them separate is what makes the claim honest:

• the stretching machinery — borrowed, 1978, everybody's
• the cost of one pairing — this is where the ball and the angle go, and this is the contribution

Move 9Averaging on a ball is not plain averaging

Once your data lives on the surface of a ball, ordinary averaging goes wrong in a specific way: the average of two points on a globe is buried inside the earth, not on the surface where the data lives.

Two lists, far apart:

first:    0.80  0.15  0.05
second:   0.05  0.15  0.80

plain average:      0.425000  0.150000  0.425000
proper midpoint:    0.403226  0.193548  0.403226

Both of those sit exactly halfway between the two, in the sense of being the same distance from each. So both look like midpoints. Here's the test that separates them — measure the whole arc, then measure it in two hops through each candidate:

the whole arc, first → second:          1.976864

through the proper midpoint:   0.988432 + 0.988432 = 1.976864     exactly the arc
through the plain average:     0.994620 + 0.994620 = 1.989240     longer

The proper midpoint's two halves add up to exactly the whole. It's on the path. The plain average sits off to one side, and going through it is a detour — small, but real, and it compounds every time you average.

The names in the papers
Walking along the surface between two points is slerp (spherical linear interpolation — "walk the arc, don't cut through"). The proper centre of a whole set of points is the Karcher mean (or Fréchet mean): the point on the surface whose total squared distance to everything else is smallest. Same idea as an ordinary average, computed on a curved surface instead of a flat one.

Move 10Why averaging a whole word goes blind

Now the payoff. Take two words that are identical almost all the way through and differ only at the very end — "cat" and "cap". Call the fraction of the word where they differ the difference stretch.

Two ways to compare them: keep the curves (Move 8), or average each word down to a single list and compare those. Watch what happens as the difference gets briefer:

Read that table down the columns. Every time the difference stretch halves:

• the curve distance halves — 0.00833 → 0.00417 → 0.00208. Fair. The difference got half as long, the measured difference got half as big.
• the averaged distance falls to a quarter — 0.00160 → 0.00040 → 0.00010. Not fair. The difference got half as long and the measurement dropped four times as much.

So averaging goes blind twice as fast as the signal actually fades. By the time the difference is a hundredth of the word, keeping the curve is 83 times more sensitive than averaging.

The name in the papers
The temporal-pooling theorem. "Pooling" is the jargon for averaging a sequence down to one summary.

Move 11Ignoring pitch

A man and a woman say the same vowel. Same vowel — but every frequency in it has moved. The engine as built so far will call them different, and it shouldn't.

The insight is about how frequency works. Going up an octave means doubling the frequency. So if you lay the buckets out on a scale where each step is a multiplication rather than an addition — the way a piano keyboard is laid out — then changing pitch just slides every bucket sideways by the same amount.

Which gives an obvious fix: try every possible slide, and keep the best match.

Why the slide costs nothing — and this is exact, not approximate

Look back at Move 3. To get the closeness number you multiply the buckets in pairs, square-root each, and add them all up.

Addition doesn't care what order you add in. 2 + 5 + 9 is the same as 9 + 2 + 5. So if you slide both lists by the same amount, you're adding up exactly the same set of numbers in a different order — and you get exactly the same total.

Here it is tested. Two random 8-bucket lists, slid together through all 8 positions:

slide     distance by angle       chi-squared      Wasserstein-1
  0      1.2319673088986400      0.2773124192     1.3864099180
  1      1.2319673088986400      0.2773124192     0.8929874312
  2      1.2319673088986400      0.2773124192     0.9436540201
  3      1.2319673088986405      0.2773124192     1.2977986517
  4      1.2319673088986400      0.2773124192     1.2598721832
  5      1.2319673088986400      0.2773124192     1.2736155293
  6      1.2319673088986400      0.2773124192     0.8929874312
  7      1.2319673088986405      0.2773124192     1.5387916435

biggest disagreement across all 8 slides:  0.00000000000000044

That last figure is 4.4 ten-thousandths of a trillionth — the last digit a computer can hold. It isn't "close". It's the same number, printed with rounding dust on it.

Now look at the third column. Wasserstein-1 jumps all over the place, because it's the one stick that reads which bucket is next to which — so wrapping the buckets round genuinely breaks it. It's therefore excluded from this part of the theory. Mentioning that exclusion is what shows you understand the mechanism instead of reciting the result.

What it buys

Measured on synthetic vowels: transposing the same vowel up by 6 buckets collapses the distance 61-fold (0.4611 down to 0.0075), while a genuinely different vowel stays far away (0.2911). That's a 39-fold separation between "same vowel, different pitch" and "different vowel" — and it repairs a misclassification the base engine gets wrong.

Plain version: it removes absolute pitch and keeps the spacing between the peaks — which is where the identity of a vowel actually lives.

Move 12The same trick, applied to time itself

Last piece, and it's the most elegant one.

Move 8's distance has a flaw: because of how the stretching works, it breaks rule 4 — the no-shortcuts rule. So strictly it's a divergence, not a fair metric. The fix is to apply the method's own founding move to the clock instead of the sound.

Here's the observation. A stretch is a schedule: go a bit slower here, a bit faster there. That schedule is made of speed-up factors that are never negative, and over the whole word they have to add up to exactly the length of the word.

Never negative. Adds to a fixed total. That's a distribution — the same object as Move 1. So square-root it, and the stretch schedule lands on a ball too.

Once you do that:

• the distance becomes a genuinely fair metric, no-shortcuts rule and all
• the padding cheat from Move 8 becomes impossible by construction — not guarded against, impossible
• the distance splits cleanly in two: how different the shapes are, and how different the timings are — and that second number turns out to be the very same angle-distance from Move 3, now measured between two schedules

Measured: it's 15 to 35 times more stable than plain stretching when both recordings are warped at once.

The name in the papers
SRVF — square-root velocity representation (Srivastava and others, 2011; the ball-shaped version, Su and others, 2014). Read it as "the square root of time".

And the closing symmetry: pitch shifts (Move 11) and time stretches (Move 12) are both things we want to ignore, and once set up this way, both are ignored by the same mechanism.

Move 13The whole thing in six lines

Everything else is a variation:

• Ignoring pitch is "slide the buckets and take the best match".
• The average of a set is "the point with the least total distance to all of them".
• SRVF is line 2 applied to the clock instead of the sound.
• The pooling result is what line 5 buys you, measured.

Glossary — the scary names, decoded