Transcription Benchmark Report
Date: 2026-11-20 · Engine: SpeechAnalyzer + SpeechTranscriber, on-device · Platform: macOS 27, Apple M5 Max · Reproduce: Scripts/bench-transcription.sh
Summary
| Locale | Voice | Audio | Reference words | S | D | I | WER | CER | Speed |
|---|---|---|---|---|---|---|---|---|---|
| en-US | Samantha | 56.0 s | 163 | 3 | 0 | 0 | 1.84 % | 0.94 % | 82× real time |
| en-US | Daniel | 59.4 s | 163 | 3 | 0 | 0 | 1.84 % | 0.94 % | 84× |
| fr-CA | Amélie | 62.5 s | 182 | 1 | 0 | 0 | 0.55 % | 0.09 % | 92× |
| fr-CA | Thomas | 68.5 s | 182 | 0 | 0 | 0 | 0.00 % | 0.00 % | 116× |
S, D, I are substitutions, deletions and insertions. Raw results, including the recognised text, are in Benchmarks/results/*.json.
These numbers are an upper bound
Read them for what they are:
- The speech is synthetic. The passages were spoken by macOS voices with
say. Synthetic speech is clean, evenly paced, close-miked and free of overlap, hesitation, accent variation and room sound. Real meetings will produce more errors, possibly many more. - Four samples, about a minute each, one passage per language. This is a sanity check that the pipeline works and scores correctly, not a statistical estimate.
- Measured on a Mac. The same API exists on iOS, but model size, speed and accuracy on an iPhone were not measured. No transcription has been run on a physical iOS device.
- Speed excludes the model download. The first run in each language waited several minutes for the system to install the speech model.
No accuracy figure is claimed for real-world recordings.
Method
- Reference text. Two passages written for this project — a short planning meeting, in English (
Benchmarks/reference/en-US.txt) and in French (Benchmarks/reference/fr-CA.txt). They are original text, free to use. - Audio.
say -v <voice> -r 170 -f <passage>produces an AIFF file per voice. Thomas is a France-French voice transcribed with the fr-CA model. - Transcription.
orix-bench transcribecallsTranscriptionEngine.transcribe, the same function the app uses. - Scoring.
TranscriptionScoringlower-cases, removes punctuation, folds typographic apostrophes, splits hyphenated words and keeps accents and elisions (so « l'heure » is one token). Word error rate is (S + D + I) / N from a Levenshtein alignment over words; character error rate is the same over characters of the normalised text.
The scorer is itself tested: identical texts score 0, a known substitution-plus-deletion scores 0.5, insertions are counted, and French normalisation is checked.
Observations
- All errors were substitutions. In English, the three were normalisation differences rather than misheard words — for example the recogniser writes "3 topics" and "1st Monday" where the reference spells the words out, and "canceled" for "cancelled". A scorer that normalised numbers and spelling variants would report a lower rate; the strict score is kept.
- Every reference word received a time range in English (163 of 163); 180–181 of 182 in French.
- Segmentation produced 16–17 segments per minute, one sentence each.
Language support
The app offers exactly the locales the system reports at run time through SpeechTranscriber.supportedLocales and, for the fallback, SFSpeechRecognizer locales with on-device support. It does not ship a fixed list, and none is asserted here. English (US) and French (Canada) were both available on the test Mac after their models were installed.
Still to measure
- Real recordings of real rooms, several speakers, both languages, on iPhone.
- Long recordings (an hour or more): speed, memory and battery.
- The legacy on-device recogniser path.
- Live transcription latency and stability.