1. Why we ran this
Our apps speak entirely on the device: a conversation partner in KoalaSpeak, and single words, meanings and example sentences in Vocabulary Builder and Vocab Wonders. A voice for these apps has to pass two different tests. It must start talking quickly in a conversation, and it must say a word on its own clearly enough for a learner to copy it, with no sentence around it to help.
Swift is a new voice in testing: a 13-million-parameter, non-autoregressive model from the StyleTTS 2 family, released under Apache-2.0. Swift · Heart sounded fast and pleasant in early listening. Before making it the default in our English apps, we measured it against every voice the apps already offer.
2. Method
- Voices: 16 voices from 5 model families the apps ship today: Swift (5 voices), Classic (an 82M-parameter model of the same family, full precision and 8-bit), Studio (a larger multi-voice model) and five voices of a small end-to-end architecture, about 60 MB each (Amy, Leah, Libby, Alan, Bryce; Libby is one speaker of a multi-speaker model).
- Runtime: the same CPU inference runtime and version that the apps embed, with the apps' own settings (silence scale, sentence splitting, sampling noise, phonemizer and lexicon). Each voice runs in its own process.
- Second run: after the first results, the five Swift voices were measured again through our own engine with the new phonemizer (next sections), on the same words, sentences, scorer, laptop and thread counts.
- Words: the 82-word list from our October 2026 listening test. 40 lemmas were sampled at random from the Vocabulary Builder levels A1–C2, plus 40 known-difficult words (colonel, queue, choir, debt, thorough, genre, pneumonia…) and 2 phrasal verbs.
- Sentences: 16 lines. They include conversation lines, every line that failed in earlier runs (a British spelling, “Oh wow”, “like”, debt/colonel/receipt in context), names, a phone number and date, and one-word lines.
- Scoring: an open speech-recognition model (base English size, 8-bit) transcribes each clip, with 0.25 s of silence padded on each side. A word counts only on an exact match after lower-casing and stripping punctuation. Sentences are scored by corpus word error rate (WER), excluding the one-word and number lines.
- Speed: each sentence was run three times and the fastest run kept. The real-time factor (RTF) is synthesis time divided by audio length. Time to first audio is the time to synthesise the first sentence of a reply, because the apps speak reply by reply, phrase by phrase. Both were measured at 2 threads (the apps' default) and 4 threads.
- Host: Apple M5 Pro laptop, CPU only. The phone run is pending, so these speeds are a relative ranking, not phone timings.
Reproducibility check: re-running the October list gave the same picture as the October listening test. Studio 83% (October: 83%), Classic 79–80% (74–82%), Amy 68% (68%).
3. Results
The table shows each voice as the apps run it today. For the five Swift voices that means the new phonemizer described in the next section; their first measurement, through the runtime's general phonemizer, is in the comparison table below.
| Voice | Words | Hard | Sentence WER | RTF | First audio | Size |
|---|---|---|---|---|---|---|
| Swift · Heart | 71% | 65% | 0.0% | 0.021 | 60 ms | 86 MB |
| Swift · Bella | 82% | 78% | 0.0% | 0.024 | 80 ms | 86 MB |
| Swift · Michael | 68% | 73% | 0.0% | 0.024 | 78 ms | 86 MB |
| Swift · Fenrir | 59% | 65% | 0.0% | 0.022 | 57 ms | 86 MB |
| Swift · Emma (UK) | 73% | 70% | 0.0% | 0.021 | 71 ms | 86 MB |
| Classic · Heart | 79% | 78% | 0.0% | 0.173 | 538 ms | 401 MB |
| Classic · Bella | 80% | 83% | 0.0% | 0.169 | 562 ms | 401 MB |
| Classic · Michael | 46% | 43% | 0.0% | 0.165 | 549 ms | 401 MB |
| Classic compact · Heart | 80% | 80% | 0.0% | 0.368 | 1090 ms | 190 MB |
| Amy | 68% | 68% | 0.0% | 0.022 | 63 ms | 81 MB |
| Leah | 70% | 68% | 1.4% | 0.022 | 50 ms | 81 MB |
| Libby | 72% | 73% | 2.1% | 0.022 | 56 ms | 97 MB |
| Alan (UK) | 60% | 50% | 0.7% | 0.021 | 62 ms | 81 MB |
| Bryce | 59% | 63% | 2.1% | 0.021 | 68 ms | 81 MB |
| Studio · Female 1 | 83% | 78% | 1.4% | 0.114 | 374 ms | 145 MB |
| Studio · Male 1 | 67% | 60% | 0.0% | 0.119 | 400 ms | 145 MB |
- Speed: Swift is as fast as the small single-speaker voices (RTF 0.021–0.024 at 4 threads, first audio 57–80 ms). Swift · Heart is about 8 times faster than Classic and about 18 times faster than Classic compact, with about a fifth of Classic's download. Every clip was finite and non-silent.
- Sentences: with the new phonemizer, all five Swift voices read the 12 scored sentences without a word error. The last misses of the first run are gone: “harbour” (heard as “harps”), “like” (heard as “lack”) and “colonel” (heard as “tunnel”).
- Isolated words: Swift · Bella now reads 82%, one word behind our best voice (Studio · Female 1, 83%) and above both Classic voices. Swift · Heart reads 71%, above Amy (68%), the voice it replaces as KoalaSpeak's first voice.
- 8-bit Classic is slower than full precision on this laptop CPU. We will check this on phones before drawing conclusions.
4. Where the word errors came from, and the fix
The runtime turns English text into phonemes with a general rule-based phonemizer. Swift was trained on phonemes from a different, lexicon-based phonemizer. Inside a sentence the voice recovered. On a bare word with no context it often did not: debt became “dead”, colonel “kennels”, thorough “sorrow”, height “how it”. Feeding the same 82 words to Heart through its own phonemizer, in a Python experiment, raised words read right from 56% to 73%.
We then built that phonemizer into our engine. Swift voices now look words up in the 177,000-entry US lexicon that ships with the voice, apply the same rules for stress, word endings, contractions, numbers, money, times and abbreviations, and pass the phonemes straight to the voice model. A sentence with a word outside the lexicon (a name, an interjection, a typo) is read through the runtime's phonemizer as before. That is 3.1% of the 3,236 English sentences in KoalaSpeak's content, and none of the 82 words or 16 sentences in this test. The download did not change.
| Swift voice | Words before → after | Hard before → after | Sentence WER before → after | RTF before → after |
|---|---|---|---|---|
| Heart | 56% → 71% | 50% → 65% | 0.7% → 0.0% | 0.032 → 0.029 |
| Bella | 61% → 82% | 58% → 78% | 2.1% → 0.0% | 0.035 → 0.030 |
| Michael | 56% → 68% | 65% → 73% | 1.4% → 0.0% | 0.030 → 0.033 |
| Fenrir | 51% → 59% | 58% → 65% | 2.1% → 0.0% | 0.032 → 0.040 |
| Emma (UK) | 62% → 73% | 60% → 70% | 0.0% → 0.0% | 0.030 → 0.031 |
- Heart in our engine (71%) is within two words of the Python experiment (73%). The lexicons differ on a few words and the voice model adds random noise.
- Speed did not change: RTF 0.021–0.024 at 4 threads, first audio 57–80 ms, a single word in 21–32 ms (median) and loading in 230–325 ms at 4 threads. Fenrir was slower in one 2-thread run (0.040) and normal at 4 threads.
- Heart still misses some words although its phonemes are now right: debt (“dead”), colonel (“kernels”), choir (“quiet”), thorough, rhythm, genre and queue. The Python experiment missed the same words, so this is a limit of the Heart voice, not of the phonemizer. Bella reads choir, thorough, rhythm and genre right.
- Some misses are shared by every voice and say more about the scorer than the voice: twenty (written “20”), the homophones aisle, hour and suite, and cough heard as “coffee”.
5. What we changed
- Done: Swift voices read their own phonemes in our engine. We set the bar before building it (isolated words at least 70%, sentence WER at most 1%), and Heart passed it at 71% and 0.0%.
- Done: in KoalaSpeak, new profiles start with Hazel, a new friend who speaks Swift · Heart. Amy keeps her current voice, and existing profiles keep the voice they chose.
- Done: Vocabulary Builder and Vocab Wonders now read words with Swift · Heart by default. The new phonemizer is in the engine all three apps share. Learners who picked another voice keep it, and Amy is still offered.
- Next: phone timings (first audio, RTF, memory) on a 2022 Android flagship and on an iPhone, and a blind listening test of naturalness, including whether Heart or Bella should be the word voice.
6. Limitations
- Intelligibility to a speech recogniser is not perceived quality. It says nothing about warmth or naturalness, and it punishes homophones (“suite” heard as “sweet”) and digits (“twenty” written as “20”) the same way for every voice.
- 82 words and 16 sentences are enough to rank voices and to catch regressions. They are not enough for small differences: on 82 words, one word is 1.2 percentage points.
- One-word lines (“Tea.”, “A.”) fail for almost every voice. The recogniser hears “T” or “Hey”, so these lines are left out of the WER.
- Speeds were measured on a laptop CPU. Phones are slower and throttle when hot, so read the speeds as a ranking, not as phone figures. The new phonemizer builds for Android and iOS but has not yet been timed on a phone.
- Naturalness was not scored automatically. The predicted-MOS model we planned to use could not run in our sandboxed tooling, so it will come from a blind listening test instead.
7. Tooling and data
All measurements were made with tools developed by Melipy AI Labs: the tts_compare bench (synthesis with the apps' settings, scoring and reports), a small command-line driver that runs a voice through our own engine exactly as the apps do, a phrase-level latency harness, and a blind listening sheet. The sheet shuffles voices for every sentence, hides their names and exports ratings. The word list, sentences, per-word transcripts and timings are kept with the benchmark so the same test can be re-run on every new voice.
compare.py synth MODELS OUT --threads 2 # one process per voice, app settings
compare.py synth MODELS OUT --threads 4
compare.py synth MODELS OUT --threads 2 --core ENGINE_BIN # Swift voices through our own engine
compare.py score MODELS OUT # speech-recognition transcripts, 0.25 s padding
compare.py report OUT result.json # summary + per-word and per-sentence data
listen.py OUT sheet/ # blind listening page + hidden key
Cite this note
Melipy AI Labs (2026). Sixteen on-device English voices: how fast, and how often a listener hears the right word. Melipy Research Notes, 5 November 2026. https://melipy.com/en/research/on-device-voices-benchmark-2026-11