Why AI Transcription Tools Make Different Mistakes

Feed the same audio file into two transcription tools and you rarely get two versions of the same errors. One tool nails a piece of medical jargon but garbles a speaker's name; the other gets the name right but loses a sentence during crosstalk. That divergence is not random. Each tool is trained on different audio, weighted toward different accents and vocabularies, and built from separate components for recognizing words, resolving ambiguity, and identifying speakers. This article walks through each of those components, explains where its errors come from using published research and vendor documentation, and gives you a way to test a tool on your own speakers rather than trust a marketing accuracy number.

Word error rate is one number hiding several different failures

Word error rate (WER) — the standard metric for scoring transcription accuracy — is calculated as the number of substitutions, deletions, and insertions divided by the total number of words in the correct reference transcript (Wikipedia, Word error rate; the formula itself derives from Levenshtein edit distance at the word level). A substitution is a wrong word swapped in for the right one. A deletion is a word the model missed entirely. An insertion is a word the transcript adds that nobody actually said.

That single number, though, collapses three structurally different problems into one score. A model with a 10% WER on a given recording could be making mostly substitution errors from accent mismatch, mostly deletion errors from a noisy microphone, or a mix of insertion errors from overlapping speech. Two tools can post the identical WER on a benchmark and still fail in completely different, non-overlapping places on your specific audio.

This matters practically because WER is usually reported as a single aggregate figure across a benchmark dataset, and benchmark datasets are themselves a choice. A tool benchmarked mostly on clean, single-speaker, standard-accent audio will report a low WER that has almost no predictive value for a noisy two-person phone call recorded in a non-standard dialect. The number is real, but the conditions it was measured under are doing most of the work.

A five-vendor study published in the *Proceedings of the National Academy of Sciences* found that automatic speech recognition systems from Amazon, Apple, Google, IBM, and Microsoft averaged a 35% word error rate on Black speakers versus 19% on white speakers reading the same structured interview questions — and the gap held even on identical phrases spoken by both groups, pointing to the acoustic model itself, not vocabulary or grammar differences (Koenecke et al., 2020, *PNAS*).

Because WER hides the error type, the rest of this article separates transcription mistakes into their actual sources: the acoustic model, the language-context model, the diarization system, the vocabulary the model was trained on, and the raw audio quality. Each is a distinct subsystem with its own failure modes, which is the real reason two tools disagree on the same recording.

Every acoustic model has an implicit baseline for "normal" speech

Speech recognition starts with an acoustic model, the component that maps raw sound waves to phonemes and words based on statistical patterns learned from training audio. Whatever accents, dialects, speaking rates, and recording conditions dominate that training set become the model's implicit definition of clearly spoken language. Audio that departs from that baseline, even if it is perfectly intelligible to a human listener, produces systematically more substitution and deletion errors.

OpenAI's Whisper, one of the most widely deployed open speech models, was trained on 680,000 hours of largely web-scraped, multilingual audio (Radford et al., 2022, "Robust Speech Recognition via Large-Scale Weak Supervision," OpenAI). The model's own published documentation reports per-language error rates that vary substantially, with lower-resource languages producing markedly higher word error rates than the handful of languages best represented in that training set. The same dynamic shows up within a single language: a model trained predominantly on North American broadcast or podcast English will structurally underperform on a Scottish, Nigerian, or Indian-accented recording of comparable audio quality, independent of how clearly the speaker is actually enunciating.

The Koenecke et al. (2020) PNAS study cited above is the clearest documented case of this at scale. Because the racial disparity in error rate persisted on identical phrases spoken by both Black and white participants, the researchers concluded the gap traced to the acoustic model's phonetic representations, not to differences in vocabulary, grammar, or dialect-specific word choice. In other words, the model had learned "what English sounds like" from a training set that underrepresented one dialect's phonetic patterns, and that gap showed up as an accuracy gap in production.

Why retraining on more data does not fully fix this

Adding more audio to a training set reduces the average error rate but does not automatically balance error rates across accents unless the added data specifically corrects the imbalance. If a company scrapes more podcast audio to expand its training set, and podcast audio still skews toward the same demographic and regional distribution as before, the model's relative blind spots persist even as its overall average score improves. Vendors that have measurably narrowed these gaps have generally done so by deliberately sourcing more diverse accent and dialect data, not simply by scaling up an already-skewed corpus.

Homophones get resolved by context, and different models weight context differently

"Their," "there," and "they're" sound identical in fluent speech. So do "to," "too," and "two," and "write" and "right." An acoustic model alone cannot distinguish these; the words are phonetically indistinguishable. Resolving them is the job of a separate language model layered on top of the acoustic output, which predicts the most statistically likely word given the surrounding sentence, the same underlying mechanism a text-generation model uses to predict the next word in a sequence (see how AI models learn from feedback for how that kind of prediction gets trained and refined).

Because different transcription vendors train their language-context layer on different text corpora, two models can make opposite calls on the exact same ambiguous audio. One model, trained more heavily on casual conversational text, might default to "there" in a sentence structure where a model trained more on formal writing would correctly infer "their." Neither model is "hearing" the audio differently; they are making different statistical bets about what a human would most likely have written in that context.

This is the same class of problem that produces confident, fluent errors elsewhere in AI systems: the model is not checking a ground truth, it is generating the statistically probable continuation given its training distribution (see why AI hallucinates facts for the parallel failure mode in text generation, where the same lack of a built-in verification step produces plausible-sounding but wrong output). In transcription, a wrong homophone call is a smaller, more contained version of the same underlying gap: fluency without verification.

How much does context window size matter here

A model that only looks at the immediate two or three surrounding words will resolve fewer homophone ambiguities correctly than one that considers the full sentence or paragraph. Longer-context resolution generally reduces homophone errors, but it also increases the chance that an early transcription mistake propagates forward and skews later context-based decisions in the same passage.

Speaker diarization is a separate model with a separate error rate

Speaker diarization — labeling which speaker said which words, sometimes summarized as answering "who spoke when" — runs as an entirely distinct process from word-level transcription. Diarization systems typically extract voice embeddings (numerical representations of vocal characteristics like pitch and timbre) and cluster them to infer speaker boundaries, independent of what words were actually said.

Diarization has its own standard evaluation metric, the diarization error rate (DER), introduced for the NIST Rich Transcription evaluations. DER sums three components divided by total reference speaker time: false alarms (speech attributed to a speaker where the reference shows none), missed speech (reference speaker time the system failed to attribute to anyone), and speaker confusion (time correctly detected as speech but assigned to the wrong speaker) (NIST Rich Transcription evaluation framework; see also the pyannote.metrics documentation for the canonical formula). Because DER and WER are computed independently, a tool can transcribe every word in a meeting correctly while still mislabeling half the turns, or vice versa.

Overlapping speech is diarization's classic failure condition. When two speakers talk simultaneously, even briefly, the voice-embedding clustering step has to decide how to split that segment, and it frequently assigns the entire overlap to whichever speaker's voice characteristics dominate the audio mix. Similar-sounding voices, such as two speakers of the same gender and similar age on a phone call, compound the problem because the embeddings the model uses to distinguish them are closer together in the first place. Quick back-and-forth exchanges, the kind common in interviews or debates, produce the same failure for a related reason: diarization models typically need a minimum segment length to confidently cluster a voice, and rapid turn-taking generates segments shorter than that threshold.

Because word recognition and diarization are architecturally separate, a transcription tool's overall accuracy reputation can be misleading for meetings specifically, where diarization errors matter as much as word errors. This is directly relevant to how AI note-taking apps summarize meetings: a summarization layer built on top of a transcript with wrong speaker labels will confidently attribute a decision or commitment to the wrong person, even if every word in the underlying transcript is correct.

Domain vocabulary is a fourth, independent failure mode

Beyond accent, homophone resolution, and diarization, a transcription model's training text corpus determines how well it recognizes specialized vocabulary: drug names, legal terms of art, product names, or industry-specific jargon. A model trained predominantly on general web text has, by construction, seen common English words far more often than a niche technical term, and when the acoustic signal for an unfamiliar term is ambiguous, the model's language layer will tend to substitute the closest common word it has seen more frequently, even when the underlying audio is perfectly clear.

A cardiologist saying "apixaban" might get transcribed as a nonsense phonetic near-miss, or silently replaced with a more common drug name that happens to share some phonemes, because the model's training data contained thousands of examples of the common name and vanishingly few of the specific one. The same dynamic applies to company and product names, legal citations, and scientific terminology: rarity in the training corpus, not audio clarity, drives the error.

This is a well-documented enough limitation that several transcription vendors now offer custom vocabulary lists, glossary uploads, or fine-tuning options specifically for specialized industries like healthcare, law, and enterprise software. That feature exists because general-purpose training data has real, predictable, and fixable blind spots for domain vocabulary, not because the underlying acoustic or language model is broadly unreliable.

Domain adaptation — the practice of supplying a model with a custom list of expected terms, names, or jargon before transcription — narrows this specific error type without touching the accent or diarization problems described above. That distinction matters when troubleshooting: if a tool mistranscribes technical terms but handles accents and speaker turns fine, the fix is a vocabulary list, not a different vendor.

Audio quality compounds every error type simultaneously

Accent handling, homophone resolution, diarization, and vocabulary recognition all implicitly assume a reasonably clean acoustic signal to begin with, and real-world recordings rarely deliver that. A laptop's built-in microphone picking up keyboard clatter, room reverberation in a large conference space, or a phone held at arm's length all degrade the signal the acoustic model has to work with before any of the downstream disambiguation steps even run.

This is why two transcription tools tested on a studio-quality recording can post nearly identical accuracy, then diverge sharply on the same speakers recorded through a laptop microphone in an echoey room. Degraded audio pushes every model closer to its individual failure points at once: the acoustic model's phoneme recognition gets noisier, which means the language model has more ambiguous input to resolve, which means diarization has less reliable voice-embedding data to cluster on. Each subsystem's error rate independently climbs, and because the systems are not equally robust to noise, the compounding does not happen at the same rate across tools.

A model trained on a narrower range of acoustic conditions, for instance mostly studio-recorded podcasts, will typically show a steeper accuracy drop-off on noisy audio than one trained on a broader mix of phone calls, field recordings, and lower-fidelity sources, even if the two models perform identically on clean audio. This is a second, independent reason accuracy benchmarks measured under ideal conditions predict real-world performance poorly: the benchmark condition and your actual recording condition may sit at very different points on each model's individual noise-robustness curve.

A field guide to what's actually going wrong

The table below maps the five error sources discussed above to their typical cause and the pipeline stage responsible, so a transcript can be diagnosed rather than just re-run on a different tool and hoped to improve.

Error type	Typical cause	Pipeline stage	Fix that actually helps
Accent/dialect substitution errors	Training audio underrepresented that accent or dialect	Acoustic model	Test on your actual speakers; some vendors let you select accent-specific model variants
Homophone confusion	Ambiguous phonetics resolved by a language-context model trained on a different text distribution	Language model (post-acoustic)	Longer context windows; manual correction of known ambiguous terms
Speaker misattribution	Overlapping speech, similar voices, or short rapid turns confuse voice-embedding clustering	Diarization (separate from word recognition)	Ask speakers to avoid talking over each other; use per-speaker microphones when possible
Domain jargon errors	Rare terms underrepresented in training text corpus	Language model / vocabulary layer	Custom vocabulary lists or glossary upload, offered by most enterprise transcription vendors
General accuracy collapse on noisy audio	Background noise, echo, or poor microphone degrades the acoustic signal	Acoustic model (input stage)	Better microphone placement; noise-cancelling hardware before the software layer, not after

Each error type traces to a distinct stage of the transcription pipeline, which is why a single accuracy score cannot predict how a tool will fail on your specific recording.

Common questions about transcription accuracy

Why does a single "accuracy" number on a vendor's website not tell you much

Because that number is an average word error rate over one benchmark dataset, recorded under one set of conditions, in one set of accents. Your recording's accent mix, background noise level, speaker overlap frequency, and vocabulary are almost certainly different from the benchmark's, and each of those factors independently shifts error rate.

Does using a more expensive or newer transcription tool fix all of these problems at once

Not necessarily. A newer model can reduce overall word error rate while leaving specific gaps, such as accent disparity or diarization on overlapping speech, largely intact, because those gaps come from training data composition and pipeline architecture rather than raw model scale. Testing the specific failure mode you care about is more reliable than assuming a newer release fixed it.

Is there a way to reduce homophone and jargon errors without switching tools

Yes. Supplying a custom vocabulary list, when the tool supports it, directly reduces domain-jargon substitution errors, and providing longer, cleaner audio context reduces homophone ambiguity because the language model has more surrounding text to weight its prediction on.

The takeaway

Transcription mistakes are not one problem with one fix. They trace back to distinct, identifiable weak points: an acoustic model's training accent baseline, a language model's context-resolution strategy, a diarization system's voice-clustering thresholds, a vocabulary gap in the training corpus, or raw audio quality degrading all of the above at once. Two tools rarely fail in the same place because each of those components was built and trained independently, which is why a published word-error-rate number is a starting point, not an answer, and testing a tool on your actual speakers and actual recording conditions remains the only reliable way to know how it will perform for you.

---

Read next:

• How AI note-taking apps summarize meetings
• Why AI hallucinates facts
• How AI models learn from feedback

FAQ

Q: Why does a transcription tool struggle with a specific person's accent?

The acoustic model's error rate reflects how well-represented similar accents and speech patterns were in its training audio. A landmark PNAS study found major ASR vendors averaged a 35% word error rate on Black speakers versus 19% on white speakers, a gap traced to the acoustic model's phonetic representations rather than vocabulary differences (Koenecke et al., 2020).

Q: How do transcription models decide between homophones like "their" and "there"?

A separate language-context model predicts the most statistically likely word given surrounding text, the same mechanism a text-generation model uses to predict the next word in a sentence. Different tools trained on different text corpora can resolve the same ambiguous audio differently.

Q: Is speaker labeling (diarization) part of the same process as word transcription?

No. Diarization typically clusters voice embeddings to guess speaker boundaries and is scored with a separate metric, diarization error rate (DER), independent of word error rate. A tool can transcribe words accurately while still misattributing which speaker said them, especially during overlapping speech.

Q: What is word error rate and how is it calculated?

Word error rate (WER) equals the number of substitutions, deletions, and insertions divided by the total words in the correct reference transcript, derived from word-level Levenshtein edit distance. It is the standard accuracy metric for transcription systems, though it averages together several distinct error types.

Q: Can adding a custom vocabulary list fix accent-related transcription errors?

No. A custom vocabulary list helps with domain-specific jargon and rare terms by giving the language layer more likely candidates to choose from, but it does not change the acoustic model's underlying phonetic training, which is what drives accent-related errors.