A public, reproducible evaluation of realtime Amharic speech recognition on held-out natural and read speech, scoring both accuracy and how fast text reaches the screen. The clips, every system's transcripts and the scorer are public at addisai/addis-scribe-benchmark, so you can run it again with your own keys.
Addis Scribe Streaming is the fastest of the systems we timed and the only one whose live text is final about 30 ms after speech ends. Two systems that need the whole recording, Hohe and Google Gemini 4 Argon, are more accurate on natural speech, and we publish that alongside the rest.
On 100 held-out WAXAL clips, streaming in 0.32 s steps, Addis Scribe records 28.4% WER and 13.1% CER. Google Chirp 3 records 32.5% WER streaming and 29.2% WER when given the whole file at once.
Read the full accuracy tablesFirst words appear after 1.56 s and the transcript is final 30 ms after speech ends. Google Chirp 3 streaming takes 6.67 s and 1,411 ms on the same clips.
Read how latency was measuredHohe, an open wav2vec2-BERT model, records 25.4% WER on the same WAXAL clips and Google Gemini 4 Argon 25.6%, 2 to 3 points ahead of Addis Scribe. Both need the full recording before they can answer, so neither can run on live audio.
Read the limitationsStreaming here means the model reads each new piece of audio once, keeps what it learned in a cache, and updates the text without re-reading earlier audio.
| System | Architecture | Live audio | How it handles live audio |
|---|---|---|---|
| Addis Scribe Streaming | Cache-aware FastConformer, hybrid RNNT + CTC, 0.6B | Native streaming | Each 0.32 s (or 1.12 s) chunk is read once and cached, so cost and delay stay flat however long someone talks. |
| Google Chirp 3 (am-ET) | Cloud API | Native (cloud API) | Server-side streaming over the network. Also tested as a batch request on the whole file. |
| Google Gemini 4 Argon | Cloud API | Offline only | Called through the Gemini API (gemini-4-argon) with the whole clip and one transcription prompt, default settings. No live mode, so scored for accuracy and processing time. |
| Google Gemini Flash (latest) | Cloud API | Offline only | Called through the Gemini API (gemini-flash-latest) with the whole clip and one transcription prompt, default settings. No live mode, so scored for accuracy and processing time. |
| Shook (Whisper medium) | Whisper medium, Addis AI's previous open ASR model | Not supported, run live | Whisper transcribes complete audio. Run live with the WhisperLiveKit method: once per second it re-transcribes all audio so far, so delay grows with length. |
| Hohe (wav2vec2-BERT) | wav2vec2-BERT, third-party open model | Offline only | Needs the whole recording before it can answer. Scored for accuracy only. |
Shook is Addis AI's previous open speech recognition model. Hohe is a third-party open model, scored for accuracy only because it cannot run on live audio. Google Gemini has no live mode, so it is scored for accuracy and processing time.
Both test sets are public and neither was used in training. WAXAL test, FLEURS and the evaluation speakers were held out of every training stage.
| Test set | Repository | Split | What it measures | Selection |
|---|---|---|---|---|
| WAXAL | google/WaxalNLP | amh test | Natural speech with human transcripts | 100 clips, 22 speakers, 29 minutes, 2 to 30 s each, drawn at random with a fixed seed |
| FLEURS | google/fleurs | am_et test | Read sentences | First 150 clips for streaming, all 516 offline |
Every system is scored with the same text normaliser: Unicode NFC, punctuation and symbols removed, whitespace collapsed. WER is word error rate and CER is character error rate, both lower is better. Addis Scribe outputs text without punctuation, so removing it from every system keeps the comparison even.
Both Gemini models were called through the Gemini API on 27 October 2026 (model ids gemini-4-argon and gemini-flash-latest) with each model's default settings, no thinking or temperature overrides: the whole clip plus one transcription prompt. Processing time is the request time, network included.
Natural conversational speech is harder than read sentences, so WAXAL error rates run higher than FLEURS for every system.
100 held-out clips, word error rate, lower is better
First 150 test clips, word error rate, lower is better
| System | Mode | WER | CER | Final text after speech ends | First words | Processing per minute of audio |
|---|---|---|---|---|---|---|
| Addis Scribe Streaming | Streaming, 0.32 s chunks | 28.4% | 13.1% | 30 ms | 1.56 s | n/a |
| Addis Scribe Streaming | Streaming, 1.12 s chunks | 27.3% | 12.4% | 34 ms | 2.21 s | n/a |
| Addis Scribe Streaming | Upload | 27.3% | 12.4% | n/a | n/a | 1.08 s |
| Google Chirp 3 (am-ET) | Streaming, realtime | 32.5% | 16.5% | 1,411 ms | 6.67 s | n/a |
| Google Chirp 3 (am-ET) | Batch, whole file | 29.2% | 13.5% | n/a | n/a | 9.74 s |
| Google Gemini 4 Argon | Whole file, default settings | 25.6% | 11.6% | n/a | n/a | 47.31 s |
| Google Gemini Flash (latest) | Whole file, default settings | 28.2% | 14.8% | n/a | n/a | 28.06 s |
| Shook (Whisper medium) | Live, WhisperLiveKit method (1 s updates) | 35.0% | 16.6% | 10,839 ms | 3.66 s | n/a |
| Shook (Whisper medium) | Offline | 35.0% | 16.6% | n/a | n/a | n/a |
| Hohe (wav2vec2-BERT) | Offline only | 25.4% | 11.2% | n/a | n/a | n/a |
Bold marks the best value in each column across every mode, including offline ones. The 1.12 s chunk setting trades about 0.65 s of extra delay for 1.1 points lower WER. Processing per minute is whole-file transcription, one clip at a time; for Google it is the full request time, network included. It was not measured for Hohe and Shook.
| System | Mode | Clips | WER | CER |
|---|---|---|---|---|
| Addis Scribe Streaming | Streaming, 0.32 s chunks | First 150 | 19.8% | 8.2% |
| Hohe (wav2vec2-BERT) | Offline | First 150 | 20.6% | 7.9% |
| Previous Addis AI streaming model | Streaming, 0.32 s chunks | First 150 | 35.0% | 17.0% |
| Addis Scribe Streaming | Offline | All 516 | 20.6% | 7.7% |
| Hohe (wav2vec2-BERT) | Offline | All 516 | 20.2% | 7.3% |
On the first 150 clips Addis Scribe streaming has the lowest WER and Hohe the lowest CER. On all 516 clips offline, Hohe leads on both, by 0.4 points of WER and CER.
WAXAL word error rate split by speaker gender, taken from the dataset's own speaker metadata.
| System | Mode | Male WER | Female WER | All WER |
|---|---|---|---|---|
| Addis Scribe Streaming | Streaming, 0.32 s chunks | 27.7% | 28.8% | 28.4% |
| Addis Scribe Streaming | Streaming, 1.12 s chunks | 27.9% | 27.0% | 27.3% |
| Google Chirp 3 (am-ET) | Streaming, realtime | 33.5% | 31.9% | 32.5% |
| Google Chirp 3 (am-ET) | Batch, whole file | 29.2% | 29.2% | 29.2% |
| Google Gemini 4 Argon | Whole file, default settings | 27.2% | 24.7% | 25.6% |
| Google Gemini Flash (latest) | Whole file, default settings | 28.6% | 27.9% | 28.2% |
| Hohe (wav2vec2-BERT) | Offline only | 26.3% | 25.0% | 25.4% |
Hohe has the lowest WER for male speakers and Google Gemini 4 Argon for female speakers. With only 8 male and 14 female speakers, gaps between the groups may reflect these speakers and recordings rather than gender itself.
Audio is fed at real-time speed, as from a live microphone. We report two numbers: how long until the first words appear, and how long after the speaker stops until the transcript is final.
WAXAL, 100 clips, lower is better
WAXAL, 100 clips, lower is better
Addis Scribe and Shook were measured on one NVIDIA A100 with no network hop. Google's figures include the network round trip to its EU endpoint from the benchmark machine, so part of its delay is transport rather than the model. Shook's bar is cut at 2 s; its delay grows with utterance length because every update re-transcribes all audio so far.
The clips, human references, every system's transcripts from our run, the scorer and a runner for each system are published at addisai/addis-scribe-benchmark. Run any system again with your own keys and see your numbers next to ours on the same 100 clips.
The 100 WAXAL test clips: ids, speakers, speaker gender from WAXAL's metadata, durations and human reference transcripts.
Every system's transcript for every clip from our run, with live latencies where they were measured.
The scorer: the shared text normaliser, WER, CER and the split by speaker gender.
One runner per system with the exact settings we used, a command-line runner and a local web app.
Published at addisai/addis-scribe-benchmark.
pip install -r requirements.txt # add requirements-open-models.txt for the open modelspython fetch_audio.py # downloads the WAXAL clips into data/audio/python app.py # open http://127.0.0.1:8787, paste your keys, press Submit
The app opens at http://127.0.0.1:8787 and runs on your computer only. It shows your WER and CER next to ours on the same clips, marks every error word by word, splits by speaker gender, and measures speed and estimated cost from your run. Your keys are sent only to each service and are never written to disk. You need Python 3.10 or later and ffmpeg.
The original run also ships with the model at addisai/addis-scribe-streaming/benchmark. Questions to contact@addisassistant.com.
Addis Scribe Streaming is published on Hugging Face with the checkpoint, a streaming helper and the benchmark records.