4 min read Field note for ChorusReader
A Pronunciation Engine of Our Own
ChorusReader now turns six languages into phonemes with its own code. The engine it replaced became the black-box referee, and the scoreboard is why I trust the result.
A neural voice does not read letters. It reads phonemes, and something has to turn “souvent” into the sounds a French speaker would make before the model hears a thing. That step is called grapheme-to-phoneme, G2P, and for ChorusReader’s Standard voices outside English it used to be someone else’s library.
As of version 3.2 it is ours. Spanish, French, Italian, Portuguese, Hindi and Mandarin now go through ChorusG2P: about 10,800 lines of Swift, a rules file per language, no dependency underneath.
I want to explain why, and then spend most of this post on the part that matters more: how I know it is not worse.
Why replace a thing that works
Two reasons, one boring and one not.
The boring one is the licence. eSpeak NG is GPL-3.0. That is a fine licence for eSpeak and the wrong one to link statically into an App Store binary. The pronunciation layer needed to be code I could ship.
The interesting one is that the old path was subtly wrong, and had been the whole time. Kokoro, the model behind the Standard voices, was trained on text phonemized with tie characters: the “ch” in Sancho is one symbol, ʧ. The app never asked eSpeak for ties, so it had been sending t and ʃ as two tokens, and “aɪ” as two vowels where the model learned one. The voice coped. But the first time I compared the new engine against the app as shipped, only 97.56% of Spanish words matched, and the entire gap was those tokens. The library was right. My call into it was not. You only find that kind of thing when you own the layer.
The referee
The risk in replacing a mature engine with your own is obvious: you ship something that feels fine to you and is worse for every reader who actually speaks Portuguese.
So eSpeak stayed, in one place only: a developer tool that never ships. It became the referee. I treated it as a black box, sound in and symbols out, and did not read its rule or dictionary files. Then:
- Public-domain prose for each language, split into a set I tuned on and a held-out set by a different author. Tuning on one writer and reporting on another is the difference between learning Spanish and memorising a book.
- Two numbers per language: the share of running words whose phonemes are identical, and the phoneme error rate.
- A third mode that turned out to matter most, which I’ll get to.
Held-out results:
| Language | Words identical | Phoneme error rate |
|---|---|---|
| Spanish | 99.85% of 46,979 | 0.11% |
| French | 95.56% of 50,381 | 1.38% |
| Hindi | 93.25% of 59,423 | 2.44% |
| Italian | 91.69% of 46,932 | 2.22% |
| Portuguese | 89.24% of 47,265 | 2.63% |
| Mandarin | 3,000 of 3,000 lines byte-identical | 0.00% |
Mandarin is a different kind of result. Kokoro’s Chinese was trained through a specific Python frontend, so the job there was not to be good, it was to be the same: a byte-for-byte port of the segmenter, the pinyin matcher and the number reader. Getting the last lines to match meant shipping CPython’s own logarithm values with the data, because the word segmenter breaks ties on sums of them and “almost the same number” picks a different word boundary.
Ninety percent is where the bugs hide
Italian went 65% → 83% → 90% → 91.7% over four rounds. Portuguese took six. And somewhere around ninety, the word-match number stops telling you anything. The remaining errors are not one big mistake. They are forty small ones, each spread thinly over words that look unrelated.
What unlocked it was grouping the misses by spelling ending. A rule that is wrong for every word in -ere is invisible at 0.3% of running text and unmissable as “this class of 60 words is 40% wrong.” The eval tool grew a mode that scores by ending and exits non-zero if any class of ten or more words is a quarter wrong. From then on a regression in a small class failed the build instead of hiding inside a good average.
Different is not the same as worse
Word-identical is a strict test and a slightly unfair one. Where ChorusG2P and eSpeak disagree, eSpeak is not always the one that’s right. It reads ”* * *” as asterisco asterisco asterisco; we skip it.
So there is a second harness that asks the only question a listener cares about. Take the sentences where the two engines disagree, render both phoneme streams through the same Kokoro model, and measure the audio: Whisper’s transcription error, the model’s own likelihood of the phonemes, spectral distance, with bootstrap intervals and a bit-identity control so the harness can’t flatter either side.
It caught something. French was measurably worse: of 172 differing sentences, 51 came out worse and 18 better. The cause was one class. Nouns and adverbs ending in -ent were being read as verbs, where the ending is silent. Souvent and torrent lost their last vowel.
After the fix: 26 worse, 16 better, no measurable difference (p = 0.16), Whisper error 2.29% against 2.37% over 300 sentences. That is the sentence in the 3.2 release notes about “souvent” and “torrent”, and it exists because a harness refused to let a 95% score stand in for a listening test.
What it is not
It is not finished, and the table says so. Italian and Portuguese agree with the referee on roughly nine words in ten; I have audio-level parity evidence for French, not yet for every language. English still runs on Misaki. The letter-to-sound tables started from Epitran’s MIT-licensed inventories, and Hindi leans on CMU Flite’s; the stress, liaison, vowel-openness and cross-word rules on top are ours.
The engine reads about 70,000 Spanish words a second, which is the least interesting number here. The interesting one is that I can now fix a mispronunciation in an afternoon, in a file I can read, and a held-out author will tell me whether I made anything else worse.
Owning a layer does not mean writing it. It means being able to measure it.