SPB Git

Repository Public License

transcription-benchmark.md 3.9 KB

# Transcription Benchmark Report

Date: 2026-11-20 · Engine: SpeechAnalyzer + SpeechTranscriber, on-device · Platform: macOS 27, Apple M5 Max · Reproduce: Scripts/bench-transcription.sh

# Summary

Locale Voice Audio Reference words S D I WER CER Speed
en-US Samantha 56.0 s 163 3 0 0 1.84 % 0.94 % 82× real time
en-US Daniel 59.4 s 163 3 0 0 1.84 % 0.94 % 84×
fr-CA Amélie 62.5 s 182 1 0 0 0.55 % 0.09 % 92×
fr-CA Thomas 68.5 s 182 0 0 0 0.00 % 0.00 % 116×

S, D, I are substitutions, deletions and insertions. Raw results, including the recognised text, are in Benchmarks/results/*.json.

# These numbers are an upper bound

Read them for what they are:

  • The speech is synthetic. The passages were spoken by macOS voices with say. Synthetic speech is clean, evenly paced, close-miked and free of overlap, hesitation, accent variation and room sound. Real meetings will produce more errors, possibly many more.
  • Four samples, about a minute each, one passage per language. This is a sanity check that the pipeline works and scores correctly, not a statistical estimate.
  • Measured on a Mac. The same API exists on iOS, but model size, speed and accuracy on an iPhone were not measured. No transcription has been run on a physical iOS device.
  • Speed excludes the model download. The first run in each language waited several minutes for the system to install the speech model.

No accuracy figure is claimed for real-world recordings.

# Method

  1. Reference text. Two passages written for this project — a short planning meeting, in English (Benchmarks/reference/en-US.txt) and in French (Benchmarks/reference/fr-CA.txt). They are original text, free to use.
  2. Audio. say -v <voice> -r 170 -f <passage> produces an AIFF file per voice. Thomas is a France-French voice transcribed with the fr-CA model.
  3. Transcription. orix-bench transcribe calls TranscriptionEngine.transcribe, the same function the app uses.
  4. Scoring. TranscriptionScoring lower-cases, removes punctuation, folds typographic apostrophes, splits hyphenated words and keeps accents and elisions (so « l'heure » is one token). Word error rate is (S + D + I) / N from a Levenshtein alignment over words; character error rate is the same over characters of the normalised text.

The scorer is itself tested: identical texts score 0, a known substitution-plus-deletion scores 0.5, insertions are counted, and French normalisation is checked.

# Observations

  • All errors were substitutions. In English, the three were normalisation differences rather than misheard words — for example the recogniser writes "3 topics" and "1st Monday" where the reference spells the words out, and "canceled" for "cancelled". A scorer that normalised numbers and spelling variants would report a lower rate; the strict score is kept.
  • Every reference word received a time range in English (163 of 163); 180–181 of 182 in French.
  • Segmentation produced 16–17 segments per minute, one sentence each.

# Language support

The app offers exactly the locales the system reports at run time through SpeechTranscriber.supportedLocales and, for the fallback, SFSpeechRecognizer locales with on-device support. It does not ship a fixed list, and none is asserted here. English (US) and French (Canada) were both available on the test Mac after their models were installed.

# Still to measure

  • Real recordings of real rooms, several speakers, both languages, on iPhone.
  • Long recordings (an hour or more): speed, memory and battery.
  • The legacy on-device recogniser path.
  • Live transcription latency and stability.
Files main
Menu

Theme