Benchmark: russian-technical-stt-v1
Scope: Russian podcasts with embedded English and IT terminology
Evaluated audio: two one-hour podcast episodes
Primary metric: LLM-judged acceptable-term rate on selected technical disagreements
Evaluation dates: 27–28 August 2026
We compared six speech-to-text systems on Russian technical podcasts containing English product names, programming terminology, acronyms, code-switching, and Cyrillic transliterations:
- Google Gemini 3.5 Transcribe
- ElevenLabs Scribe v2
- OpenAI GPT Transcribe
- Qwen3 ASR 1.7B
- OpenAI Whisper Large v3
- Deepgram Nova-3
A conventional WER benchmark was not possible because no complete trusted reference transcript existed. More importantly, ordinary WER does not reflect the practical importance of technical terms: confusing Rustler with Raster or omitting ASN.1 matters more to a technical reader than many filler-word differences.
We therefore evaluated the systems on a targeted technical-terminology challenge set. Candidate cases came from disagreements among the six transcripts. GPT-5.6 Sol selected atomic technical disagreements without assigning the answer. Gemini 3.7 Flash then judged anonymous alternatives using local transcript context.
Before using this process without humans, we calibrated the Gemini judge against 18 blind human-annotated technical cases. It included the human answer in 94.4% of cases and reproduced human-derived provider scores with correlation 0.983. We then applied the same automated protocol to a second podcast with no human annotation.
The observed results support a leading tier of Gemini and ElevenLabs, not a statistically conclusive universal winner:
- Choose Gemini 3.5 Transcribe when maximizing technical-term coverage is the priority; it had the highest observed acceptable-term score on both episodes.
- Choose ElevenLabs Scribe v2 when canonical, publication-ready spelling matters most; it had the highest cross-episode canonical score and remained close to Gemini on acceptable terms.
- Choose GPT Transcribe when an existing OpenAI integration is valuable enough to accept a clear observed quality gap from the leading pair.
- Choose Qwen3 ASR when low reported API cost matters more than top-tier terminology quality.
- Deepgram was the fastest in aggregate request time here, but its technical-term score was substantially lower.
The intervals for Gemini and ElevenLabs overlap. The report therefore describes Gemini as having the highest observed score, rather than proving that it is universally superior.
| Rank | Provider | Automated acceptable-term score | Automated canonical-form score |
|---|---|---|---|
| 1 | Google Gemini 3.5 Transcribe | 93.1% | 75.0% |
| 2 | ElevenLabs Scribe v2 | 88.2% | 77.8% |
| 3 | OpenAI GPT Transcribe | 73.6% | 68.8% |
| 4 | Qwen3 ASR 1.7B | 64.6% | 36.8% |
| 5 | Whisper Large v3 | 56.2% | 31.2% |
| 6 | Deepgram Nova-3 | 29.2% | 18.8% |
Gemini had the highest observed acceptable-term score. ElevenLabs had the highest canonical-form score. The top three providers and the last-place provider were stable across both episodes.
These are LLM-judged targeted challenge-set scores, not verified population accuracy or population WER.
The goal was to build a repeatable comparison of STT systems for a specific production problem:
How reliably does each system transcribe important English and IT terminology embedded in Russian speech?
The benchmark addresses four questions:
- Which provider most often produces a faithful rendering of the spoken technical term?
- Which provider most often produces the canonical written form?
- Can an LLM judge reproduce human-derived provider rankings well enough to remove routine human evaluation?
- Does the ranking replicate on a second technical domain?
The benchmark does not attempt to estimate general Russian WER, diarization quality, punctuation quality, or verbatimness.
We collected six candidate Russian technical podcast episodes and reproducibly prepared the first hour of each source as mono 16 kHz AAC at 48 kbps. No loudness normalization was applied, and intros, ads, and other material occurring within the first hour were retained. Two episodes were selected for benchmark v1 because they contain dense but different technical vocabularies.
Podlodka Podcast #487 — “Как Elixir стал актуален благодаря AI в 2026”
Topics include Elixir, Erlang, BEAM, Phoenix, LiveView, Rustler, NIF, ONNX and related tools.
- Frozen run:
podlodka-487-raw-v2 - Source SHA-256:
a870431dae3be016f50756d94efe8967831a0e78df258ac7b97f53927b73939e - Role in methodology: human calibration and automated scoring
OrgProg #78 — “Эволюция баз данных: SQL, NoSQL и доминирование PostgreSQL”
Topics include MySQL, PostgreSQL, Tarantool, Scylla, NoSQL, OLTP, Couchbase, Aerospike, BSON and related database concepts.
- Frozen run:
orgprog-78-raw-v1 - Source SHA-256:
c66527a7463c4e2caa6d4069f948d7e37de12b6b83ca78bd5bef90991c93f4c9 - Role in methodology: fully automated replication
Each hour was split into 12 contiguous chunks near five minutes long. All providers received the same chunk audio.
The segmentation protocol:
- uses speech-activity estimates to prefer low-speech boundary points;
- selects boundaries jointly across the hour;
- preserves the complete source without gaps or overlaps;
- records source, chunk boundaries, durations, and SHA-256 hashes.
Total benchmark material:
2 episodes
120 minutes
24 common chunks
The available YouTube captions were retained only as quarantined source material. They were never used as evaluation truth.
| Provider ID | Route | Episode coverage |
|---|---|---|
elevenlabs/scribe-v2 |
ElevenLabs API | 24/24 chunks |
openai/gpt-transcribe |
OpenRouter | 24/24 |
deepgram/nova-3 |
OpenRouter | 24/24 |
openai/whisper-large-v3 |
OpenRouter | 24/24 |
google/gemini-3.5-transcribe |
Vercel AI Gateway | 23/24 |
qwen/qwen3-asr-1.7b |
OpenRouter | 24/24 |
Gemini missed one chunk in the first frozen run because of the gateway's free-tier restriction. It completed the full second episode.
Where supported, requests used a fixed Russian language hint. Each provider response was stored with status, transcript, latency, usage, cost, error details, model identity, and audio provenance.
Every provider transcribed the same audio chunks independently. No provider output was treated as the reference.
The transcripts were aligned to locate local spans where systems produced materially different text. Differences limited to casing, punctuation, or equivalent formatting were not sufficient by themselves.
Each candidate contained:
- the chunk and time range;
- short context before and after the disputed span;
- one local hypothesis per provider;
- provider identities hidden from downstream selection and judging.
The candidate generator preserves multi-token expansions and omissions so that co-occurring terms are not incorrectly presented as mutually exclusive alternatives.
GPT-5.6 Sol acted as a case curator, not an answer key.
It received the deterministic candidate pool and selected cases under a versioned policy. A case was eligible only if the disputed span itself represented a technical entity, such as a:
- language, runtime, framework, library, or product;
- protocol, standard, API, format, or command;
- acronym, identifier, algorithm, or established specialized phrase.
The curator had to reject:
- generic Russian lexical differences;
- fillers and boundary artifacts;
- punctuation-only differences;
- cases combining several independent decisions;
- duplicated terms without additional evaluation value;
- cases where an omission could falsely make two co-occurring terms look mutually exclusive.
The curator could only select and narrow cases. It could not mark any provider correct.
The deterministic candidate universe contained 1,508 candidates for the first episode and 1,845 for the second. Each complete pool was submitted in one curator request. The prompt instructed the curator to remove duplicate terms and distribute useful cases across the source, but the final selection was not a random or statistically representative sample.
The first episode contributed 18 technical cases used for human calibration. The second contributed 24 technical-only cases. Total scored cases: 42.
Gemini 3.7 Flash acted as the routine evaluator.
For each case it received:
local transcript context with [TARGET]
anonymous candidate labels and texts
It did not receive:
- provider identities;
- human annotations;
- a declared reference answer;
- audio in the routine condition.
It returned:
- all acceptable candidate labels;
- labels using canonical technical spelling;
- canonical text;
- confidence;
- an uncertainty indication and short rationale.
Candidate order was deterministically permuted to reduce positional bias.
For each provider and case:
- acceptable-term score gives credit for any judge-accepted representation of the intended term, including valid transliteration;
- canonical-form score gives credit only when the provider's option is among the judge's canonical labels.
Examples of potentially acceptable variants include:
Erlang / Эрланг
Rustler / Растлер
F# / F sharp / эф-шарп
This distinction avoids penalizing a recognizer merely for using Cyrillic while still measuring publication-quality spelling.
Scores are reported separately for each episode. The headline score is the arithmetic mean of the two episode accuracies, so each domain contributes equally regardless of its number of cases.
A pooled case score is intentionally not the primary aggregate because the second episode contains more cases.
For each episode/provider score, we computed a deterministic case-level bootstrap interval using 5,000 resamples. These are within-challenge case-resampling intervals only. They do not account for episode selection, LLM case selection, judge uncertainty, related terms, or future upstream model changes, and must not be interpreted as population confidence intervals for Russian technical speech.
We also report a high-confidence view using judge confidence ≥0.75. Cases below that threshold are treated as abstentions in that view; they are not sent to humans.
Human evaluation was used once to validate the judge, not as a routine pipeline stage.
A human annotator listened to the first episode's audit clips with provider identities hidden. Multiple labels could be selected when several renderings faithfully represented the same spoken term.
The 18 technical cases used for calibration were all marked clear and high confidence.
| Metric | Gemini 3.7 Flash text-only |
|---|---|
| Human answer included | 94.4% |
| Acceptable-label F1 | 0.833 |
| Exact acceptable-label set | 61.1% |
| Correlation with human-derived provider scores | 0.983 |
Exact-set agreement is stricter than the ranking objective: the judge often accepted additional equivalent transliterations that the human did not select. The near-perfect provider-score correlation showed that this disagreement had little effect on comparative ranking.
We separately tested text-only, audio-only, and audio-plus-context judges. The text-only Gemini model gave the strongest combination of:
- agreement with human-derived provider scores;
- acceptable-label F1;
- low cost;
- reliable structured output.
Audio judges did not improve the ranking signal in this calibration. Therefore, the routine benchmark uses one text-only judge and explicitly limits its claim to practical terminology quality rather than independent acoustic truth.
We repeated all 42 judgments twice with new deterministic option permutations.
| Runs compared | Exact acceptable-label-set agreement |
|---|---|
| primary vs repeat 1 | 92.9% |
| primary vs repeat 2 | 88.1% |
| repeat 1 vs repeat 2 | 88.1% |
| all three identical | 85.7% |
The provider ranking remained practically stable. Routine evaluation uses one pass; repeated passes are an optional diagnostic.
| Rank | Provider | Acceptable-term score | Canonical-form score |
|---|---|---|---|
| 1 | Google Gemini 3.5 Transcribe | 93.1% | 75.0% |
| 2 | ElevenLabs Scribe v2 | 88.2% | 77.8% |
| 3 | OpenAI GPT Transcribe | 73.6% | 68.8% |
| 4 | Qwen3 ASR 1.7B | 64.6% | 36.8% |
| 5 | Whisper Large v3 | 56.2% | 31.2% |
| 6 | Deepgram Nova-3 | 29.2% | 18.8% |
| Provider | Accepted | Score (95% within-challenge bootstrap interval) | Canonical | High-confidence score |
|---|---|---|---|---|
| Gemini 3.5 Transcribe | 17/18 | 94.4% (83.3–100%) | 83.3% | 15/16 (93.8%) |
| ElevenLabs Scribe v2 | 16/18 | 88.9% (72.2–100%) | 72.2% | 14/16 (87.5%) |
| GPT Transcribe | 13/18 | 72.2% (50.0–88.9%) | 66.7% | 11/16 (68.8%) |
| Qwen3 ASR | 12/18 | 66.7% (44.4–88.9%) | 44.4% | 10/16 (62.5%) |
| Whisper Large v3 | 9/18 | 50.0% (27.8–72.2%) | 33.3% | 8/16 (50.0%) |
| Deepgram Nova-3 | 6/18 | 33.3% (11.1–55.6%) | 16.7% | 4/16 (25.0%) |
The cases included CommonJS, ECMAScript, BEAM, Ecto Query Builder, LINQ, C#, ASN.1, PyTorch, BERT, ONNX, Livebook, Gleam, Rustler, NIF, Quickbeam, Volt, and VibeKit.
| Provider | Accepted | Score (95% within-challenge bootstrap interval) | Canonical | High-confidence score |
|---|---|---|---|---|
| Gemini 3.5 Transcribe | 22/24 | 91.7% (79.2–100%) | 66.7% | 18/20 (90.0%) |
| ElevenLabs Scribe v2 | 21/24 | 87.5% (75.0–100%) | 83.3% | 17/20 (85.0%) |
| GPT Transcribe | 18/24 | 75.0% (58.3–91.7%) | 70.8% | 15/20 (75.0%) |
| Whisper Large v3 | 15/24 | 62.5% (41.7–83.3%) | 29.2% | 11/20 (55.0%) |
| Qwen3 ASR | 15/24 | 62.5% (41.7–79.2%) | 29.2% | 13/20 (65.0%) |
| Deepgram Nova-3 | 6/24 | 25.0% (8.3–41.7%) | 20.8% | 2/20 (10.0%) |
The cases included Hexlet, MySQL, Tarantool, Picodata, Teradata, Exadata, Scylla, NoSQL, OLTP, PostgreSQL, blob store, key-value, Foursquare, CQL/SQL, Couchbase, Aerospike, ANSI, joins, CedarDB, Firebird, CPU-bound, MCP/gRPC, and BSON.
Both episodes produced the same top three:
1. Gemini
2. ElevenLabs
3. GPT Transcribe
Deepgram was last in both. Qwen and Whisper were less clearly separated and tied on the database episode.
The case-level evidence shows qualitatively different failure modes:
- Omissions: Deepgram omitted MySQL,
blob store, ANSI, and CPU-bound in selected cases; Gemini omitted OLTP and BSON on the database episode. - Phonetic approximations: examples include
pitorchfor PyTorch,tarantulfor Tarantool,coachbasefor Couchbase, andairspikefor Aerospike. - Familiar-word substitutions: Whisper produced
birdfor BERT andpythonwhere the context indicated BSON. - Identifier corruption: systems produced
onxx,o n x x, and similar forms for ONNX;asi.т,esi.1, andasa.1for ASN.1. - Valid but noncanonical transliteration:
Растлер,Эрланг,форсквер, and similar Cyrillic forms often remained acceptable while receiving no canonical-form credit. - Multiword truncation or identity substitution: examples include
Acta Query Builderfor Ecto Query Builder andxatafor Exadata.
These patterns explain why acceptable-term and canonical-form scores answer different product questions.
Provider-reported STT totals across the two episodes:
| Provider | Reported cost | Aggregate request-seconds |
|---|---|---|
| Qwen3 ASR | $0.0540 | 463.9 |
| Whisper Large v3 | $0.1130 | 184.8 |
| Gemini 3.5 Transcribe | $0.3478 | 177.7 |
| Deepgram Nova-3 | $0.5160 | 65.5 |
| GPT Transcribe | $0.5409 | 264.1 |
| ElevenLabs Scribe v2 | not reported | 388.3 |
ElevenLabs responses did not expose cost; “not reported” must not be interpreted as free.
The primary plus two repeat judge passes cost $0.5048 for 126 stored judgments. Routine evaluation uses one pass. GPT-5.6 Sol curation of the second 24-case audit cost $2.5719 and is performed once per frozen audit.
Costs reflect gateway-reported values at the time of the experiment, not universal list prices.
Gemini produced the highest observed acceptable-term score on both episodes. Its canonical score was lower than ElevenLabs on the database episode because it sometimes omitted a term or used a noncanonical rendering despite identifying most terms acceptably. The overlapping intervals mean the benchmark does not establish a statistically conclusive universal advantage over ElevenLabs.
ElevenLabs was consistently close to Gemini and had the strongest canonical spelling overall. It is the best result when publication-ready technical typography is prioritized.
GPT Transcribe was consistently third. It handled most common technical vocabulary well but made more identity substitutions than the two leaders.
Both were substantially behind the top three. Qwen had a stronger acceptable-term score on the Elixir episode, while the systems tied on the database episode. Their canonical-form scores were much weaker than their acceptable-term scores.
Deepgram was fastest by aggregate request time but had the weakest technical-term score. Its errors frequently produced plausible phonetic approximations, omissions, or ordinary-word substitutions rather than canonical technical entities.
The frozen benchmark definition records:
- evaluation dates and requested provider model IDs/routes;
- curator and judge model/prompt versions;
- confidence threshold;
- source audio and configuration hashes;
- run and audit identifiers.
The proprietary model IDs were mutable service aliases at evaluation time unless a gateway exposed a pinned upstream version. Repeating the benchmark later may therefore exercise updated provider implementations even with identical local configuration.
The automated workflow is:
uv run stt-eval evaluate \
--episode <episode-id> \
--run-id <stable-run-id> \
--audit-version <new-audit-version> \
--max-parallel 4 \
--output artifacts/<episode>-report.mdIt executes:
chunk → transcribe → align disagreements → curate → judge → score → report
Completed transcription, curation, and judgment stages are resumable. Structured records and provenance are stored in SQLite; audio remains external and is referenced by hash.
The publication bundle includes:
- this methodology and results report;
- a readable case-level appendix;
- aggregate result CSV;
- compact chunk boundary/hash CSV;
- frozen reproduction configuration;
- curator and judge protocols.
The implementation source is not public and is available on request. The published artifacts expose the data, prompts, configurations, hashes, case decisions, and aggregate calculations needed to inspect the claims, but independent end-to-end execution requires requesting the source.
- Targeted challenge set: results apply to LLM-selected technical disagreements, not all words or general Russian speech.
- Observed, automated scores: the second episode has no human/acoustic answer key; its scores are Gemini-judged technical plausibility, not verified transcription accuracy.
- Two episodes: the domains differ, but the sample remains small and within-challenge intervals overlap for the leading systems.
- Text-only judge: it measures technical plausibility and rendering quality, not independent acoustic fidelity.
- Shared language priors: the judge may favor familiar terms also preferred by language-model-heavy STT systems.
- Curator selection: the LLM determines which disagreements enter the challenge set, so the cases are not a random sample.
- One human calibrator: ranking calibration was strong, but definitive case-level ground-truth replacement was not demonstrated; exact label-set agreement was 61.1%.
- Gateway and model mutability: routing, model implementations, prices, and availability can change.
- No population WER claim: these scores do not measure punctuation, diarization, filler preservation, or general transcript completeness.
- Private implementation: source is available on request rather than through a public repository.
On this two-episode Russian technical-speech challenge set:
- Gemini 3.5 Transcribe had the highest observed acceptable-term score;
- ElevenLabs Scribe v2 was a close second and had the highest canonical-form score;
- GPT Transcribe ranked consistently third;
- Qwen3 ASR and Whisper Large v3 formed a weaker middle tier;
- Deepgram Nova-3 ranked last despite low latency.
Methodologically, the experiment shows that a disagreement-based challenge set can be curated and judged automatically after a one-time human calibration. The resulting automated ranking replicated across a second podcast and removed routine human labeling from the evaluation workflow.
The claim is deliberately narrow:
A calibrated LLM-based evaluator can compare the practical technical-terminology quality of STT systems on Russian podcasts without routine human annotation.
It does not claim that a text LLM establishes acoustic ground truth.
- Radford, A. et al. Robust Speech Recognition via Large-Scale Weak Supervision. 2022. arXiv:2212.04356
- Shi, X. et al. Qwen3-ASR Technical Report. 2026. arXiv:2601.21337
- Ma, R. et al. Can Generative Large Language Models Perform ASR Error Correction? 2023. arXiv:2307.04172
- Fiscus, J. Recognizer Output Voting Error Reduction (ROVER). 1997. NIST
- Huang, R. et al. ConEC: Earnings Call Dataset with Real-world Contexts for Benchmarking Contextual Speech Recognition. LREC-COLING 2024. ACL Anthology
- Li, Y. et al. CB-Whisper: Contextual Biasing Whisper Using Open-Vocabulary Keyword-Spotting. LREC-COLING 2024. ACL Anthology
- Morris, A. et al. From WER and RIL to MER and WIL: Improved Evaluation Measures for Connected Speech Recognition. 2004. ISCA Archive
- JiWER. GitHub
kaldialign. GitHuberror-align. GitHub