Time
“The appointment is at 8:30 p.m.”
DeafBench
I test what an ASR system drops when the detail matters.
I wrote DeafBench because a transcript can look mostly right and still lose a time, a username, a dosage, or the word “not.” It keeps ordinary word error rate, but it also scores those failures directly and records the exact evidence behind a result.
Evidence boundary
Synthetic accessibility stress demo
This demonstration uses DeafBench’s authorized synthetic corpus. It tests whether model output preserves declared critical details while audio is changed in controlled ways. It does not measure Deaf or hard-of-hearing speakers, dysarthric speech, accents, age groups, or any other human demographic.
Those questions need authorized, consented, appropriately licensed human speech, declared cohorts, sample sizes, uncertainty, and disjoint development and evaluation sets. I will not substitute a synthetic voice effect for that evidence.
What gets stressed
The details and conditions are declared before a model runs.
Every row below describes a failure DeafBench can count separately from overall WER. The full frozen suite contains 24 utterances and gives each clean reference a controlled stress condition.
Date
“The deadline is November 14.”
Dosage
“Take 12 milligrams at noon.”
Name
“Ask for Nia Solano at reception.”
Username
“The username is kai.jones_27.”
Code
“The access code is H7Q9.”
Confirmation number
“The confirmation number is 804193.”
Wi-Fi name
“Connect to North Hall Guest.”
Address
“Meet at 214 Cedar Avenue.”
Money
“The total is $46.20.”
Negation
“Do not take the west entrance.”
Noise: street, office chatter, keyboard, breathing, rustling, and wind at −5, 0, 10, and 20 dB SNR.
Signal changes: 8 kHz telephony, reverberation, long intra-phrase pauses, and speaking-rate variation.
Tracked failures: substitutions, insertions, deletions, hallucinations in interstitial noise, critical-entity misses, and caption timestamp drift over 500 ms.
Comparable lane
Synthetic-v2 results
These nine rows are recorded local observations from the same 25-sample corpus. Their metadata manifests are byte-stable, but the sample-level run artifacts are not published in the repository, so the values cannot be independently recomputed from a checkout. Canonical recall accepts harmless representation differences but does not fuzzy-match usernames, codes, Wi-Fi names, incomplete numbers, or wrong times.
| Model | WER | Strict recall | Canonical recall | Local RTFx | Peak VRAM |
|---|---|---|---|---|---|
| Qwen3-ASR 0.6B | 26.6% | 67.7% | 90.3% | 10.47 | 1.52 GiB |
| Qwen3-ASR 1.7B | 21.0% | 67.7% | 91.9% | 9.89 | 3.86 GiB |
| Parakeet TDT 0.6B v2 | 22.7% | 64.5% | 91.9% | 71.23 | 4.67 GiB |
| Granite Speech 4.1 2B | 18.9% | 69.4% | 91.9% | 12.91 | 4.35 GiB |
| Granite Speech 4.1 2B NAR | 40.6% | 64.5% | 87.1% | 45.07 | 4.26 GiB |
| ARK-ASR 0.6B | 30.1% | 66.1% | 90.3% | 14.85 | 2.20 GiB |
| ARK-ASR 0.6B INT8 ONNX | 26.6% | 66.1% | 90.3% | 2.40 | CPU |
| Distil-Whisper large-v3 | 23.8% | 66.1% | 91.9% | 0.80 | CPU |
| Whisper-AT medium.en | 26.2% | 67.7% | 96.8% | 7.34 | 4.46 GiB |
Local RTFx and VRAM depend on the test computer. This is a synthetic accessibility lane, not a Hugging Face leaderboard result and not a ranking across unlike datasets. Read the ASR evaluation methodology for the scoring and aggregation contract.
Complete inventory
All eleven integrated model paths
“Integrated” means DeafBench has an adapter or runner. It does not mean every model has comparable completed evidence. Missing results stay missing.
| Model | Evidence currently shown | Lane |
|---|---|---|
| OpenAI Whisper turbo | Completed historical reports | Core v1 and non-speech v1 |
| Faster-Whisper small.en | Completed frozen and development baselines | Core v1 and public real-speech dev |
| Whisper-AT medium.en | Recorded local observation | Synthetic-v2, real-speech smoke, and non-speech-v1 |
| Distil-Whisper large-v3 | Recorded local observation | Synthetic-v2 and real-speech smoke |
| Qwen3-ASR 0.6B | Recorded local observation | Synthetic-v2 and real-speech smoke |
| Qwen3-ASR 1.7B | Recorded local observation | Synthetic-v2 and real-speech smoke |
| Parakeet TDT 0.6B v2 | Recorded local observation | Synthetic-v2 and real-speech smoke |
| Granite Speech 4.1 2B | Recorded local observation | Synthetic-v2 and real-speech smoke |
| Granite Speech 4.1 2B NAR | Recorded local observation | Synthetic-v2 and real-speech smoke |
| ARK-ASR 0.6B | Recorded local observation | Synthetic-v2 and real-speech smoke |
| ARK-ASR 0.6B INT8 ONNX | Recorded local observation | Synthetic-v2 and real-speech smoke |
Real-speech goal
The target is the Open ASR Leaderboard contract, not a convenient local score.
DeafBench has a separate official-compatible real-speech path and a deterministic 100-sample public development cohort. The current Faster-Whisper small.en development baseline is 3.56% normalized WER on that cohort. It is development evidence, not the seven-dataset leaderboard macro-average.
The long-term goal is to beat the published 5.37% average WER reference under the official dataset revisions, preprocessing, normalization, macro-average, and submission rules. I will not claim that result unless Hugging Face verifies it.
Customer-run audit
The audit is designed to run on the customer’s computer.
Audio stays with the customer. Exported evidence is limited to sanitized aggregate metrics, environment-dependent labels, model and evaluator identifiers, configuration, counts, and artifact hashes. Raw audio, transcripts, filenames, paths, identities, secrets, and critical-information values are excluded by a fail-closed scanner.
Status: the workflow is still being validated. I am not accepting customer audio or payment yet.