Repository Public License
1<!--2ORIX Audio3File: docs/reports/transcription-benchmark.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->78# Transcription Benchmark Report910**Date:** 2026-11-20 · **Engine:** `SpeechAnalyzer` + `SpeechTranscriber`, on-device · **Platform:** macOS 27, Apple M5 Max · **Reproduce:** `Scripts/bench-transcription.sh`1112## Summary1314| Locale | Voice | Audio | Reference words | S | D | I | WER | CER | Speed |15|---|---|---|---|---|---|---|---|---|---|16| en-US | Samantha | 56.0 s | 163 | 3 | 0 | 0 | **1.84 %** | 0.94 % | 82× real time |17| en-US | Daniel | 59.4 s | 163 | 3 | 0 | 0 | **1.84 %** | 0.94 % | 84× |18| fr-CA | Amélie | 62.5 s | 182 | 1 | 0 | 0 | **0.55 %** | 0.09 % | 92× |19| fr-CA | Thomas | 68.5 s | 182 | 0 | 0 | 0 | **0.00 %** | 0.00 % | 116× |2021S, D, I are substitutions, deletions and insertions. Raw results, including the recognised text, are in `Benchmarks/results/*.json`.2223## These numbers are an upper bound2425Read them for what they are:2627- **The speech is synthetic.** The passages were spoken by macOS voices with `say`. Synthetic speech is clean, evenly paced, close-miked and free of overlap, hesitation, accent variation and room sound. Real meetings will produce more errors, possibly many more.28- **Four samples, about a minute each, one passage per language.** This is a sanity check that the pipeline works and scores correctly, not a statistical estimate.29- **Measured on a Mac.** The same API exists on iOS, but model size, speed and accuracy on an iPhone were not measured. No transcription has been run on a physical iOS device.30- **Speed excludes the model download.** The first run in each language waited several minutes for the system to install the speech model.3132No accuracy figure is claimed for real-world recordings.3334## Method35361. **Reference text.** Two passages written for this project — a short planning meeting, in English (`Benchmarks/reference/en-US.txt`) and in French (`Benchmarks/reference/fr-CA.txt`). They are original text, free to use.372. **Audio.** `say -v <voice> -r 170 -f <passage>` produces an AIFF file per voice. Thomas is a France-French voice transcribed with the fr-CA model.383. **Transcription.** `orix-bench transcribe` calls `TranscriptionEngine.transcribe`, the same function the app uses.394. **Scoring.** `TranscriptionScoring` lower-cases, removes punctuation, folds typographic apostrophes, splits hyphenated words and keeps accents and elisions (so « l'heure » is one token). Word error rate is (S + D + I) / N from a Levenshtein alignment over words; character error rate is the same over characters of the normalised text.4041The scorer is itself tested: identical texts score 0, a known substitution-plus-deletion scores 0.5, insertions are counted, and French normalisation is checked.4243## Observations4445- All errors were substitutions. In English, the three were normalisation differences rather than misheard words — for example the recogniser writes "3 topics" and "1st Monday" where the reference spells the words out, and "canceled" for "cancelled". A scorer that normalised numbers and spelling variants would report a lower rate; the strict score is kept.46- Every reference word received a time range in English (163 of 163); 180–181 of 182 in French.47- Segmentation produced 16–17 segments per minute, one sentence each.4849## Language support5051The app offers exactly the locales the system reports at run time through `SpeechTranscriber.supportedLocales` and, for the fallback, `SFSpeechRecognizer` locales with on-device support. It does not ship a fixed list, and none is asserted here. English (US) and French (Canada) were both available on the test Mac after their models were installed.5253## Still to measure5455- Real recordings of real rooms, several speakers, both languages, on iPhone.56- Long recordings (an hour or more): speed, memory and battery.57- The legacy on-device recogniser path.58- Live transcription latency and stability.59