Speech-to-Text skill for Claude Code
Transcripts measured by WER on your own audio. Claude builds transcription pipelines that hold up on real calls and meetings. It normalizes audio with ffmpeg, runs VAD so silence does not become invented sentences, picks a Whisper model and compute type or a hosted API, aligns word timestamps, attributes speakers with pyannote or per-channel splitting, and measures WER and CER on human references with a bundled script.
- name
speech-to-text - size190 lines · ≈2,150 tokens
- files2
- licenseMIT
- updated
What Claude sees before loading it
This description is the only part of the skill that sits in context all the time. Claude reads it to decide when to load the full playbook below.
description: Builds speech-to-text pipelines with Whisper, faster-whisper, WhisperX, and hosted transcription APIs - audio normalization, voice activity detection and chunking, model and compute-type choice, word timestamps and forced alignment, speaker diarization with pyannote, custom vocabulary, subtitle output (SRT/VTT), and word error rate measured on your own audio. Use when transcribing calls, meetings, podcasts, interviews, or voice notes, when adding captions, when transcripts hallucinate, drift out of sync, or mislabel speakers, or when choosing between self-hosted and API transcription.Try asking
- Transcribe our recorded sales calls with speaker labels and store them as JSON
- Whisper keeps adding 'thanks for watching' to silent parts of our meetings. Fix it
- Compare faster-whisper and two transcription APIs on our support call audio
Use it when
- Transcribing calls, meetings, podcasts, interviews, or voice notes
- Generating SRT or VTT captions with accurate timing
- Transcripts hallucinate, drift out of sync, or mislabel speakers
- Choosing between self-hosted Whisper and a transcription API
Skip it when
- You need to generate speech from text — use text-to-speech
- You need text from scanned documents rather than audio — use ocr-scans
How to check the result
Run scripts/wer.py on 30-60 minutes of human-transcribed reference audio and spot-check caption timing and speaker turns by ear on a few clips.
The common mistake it prevents
Skipping voice activity detection, so silence and hold music turn into repeated or invented sentences in the transcript.
A transcription pipeline is good when its word error rate (WER) is measured on 30-60 minutes of your own audio with human reference transcripts, timestamps line up within a few hundred milliseconds, speakers are attributed correctly where that matters, and silence or music never turns into invented sentences. Benchmarks on clean audiobooks do not predict call-center or meeting audio.
Ground rules
- Normalize audio first: mono, 16 kHz, 16-bit PCM WAV is what Whisper-family models consume internally. Decode once with ffmpeg; do not trust each library’s resampler.
- Run voice activity detection (VAD) before decoding. Most Whisper hallucinations (“Thanks for watching!”, repeated phrases) come from silence and music.
- Measure WER on your domain before and after every change, with the same normalization.
- Recordings of people are personal data. Get consent where the law requires it, define retention, and do not ship audio to an API the user has not approved.
Choose the engine
| Situation | Choice |
|---|---|
| Self-hosted, GPU available, batch files | faster-whisper (CTranslate2) with large-v3 or large-v3-turbo |
| Need word-level timestamps and diarization | WhisperX (faster-whisper + forced alignment + pyannote) |
| CPU-only or edge | faster-whisper int8 with small/medium, or whisper.cpp |
| Real-time streaming captions, low latency | A streaming API (most hosted STT providers offer WebSocket streaming); Whisper is not natively streaming |
| No infrastructure, fast start, compliance offered by vendor | Hosted transcription API; compare 2-3 on your audio |
| Many languages, code-switching | Whisper large models; test each language separately |
large-v3-turbo decodes several times faster than large-v3 with a small accuracy
cost; measure on your audio before choosing.
Workflow
- [ ] 1. Build a reference set (30-60 min, human transcripts)
- [ ] 2. Normalize audio with ffmpeg
- [ ] 3. VAD and chunking
- [ ] 4. Transcribe with chosen model and settings
- [ ] 5. Align timestamps and diarize if needed
- [ ] 6. Post-process (vocabulary, punctuation, formatting)
- [ ] 7. Measure WER/CER and timing; inspect worst segments
- [ ] 8. Output in the needed formats1. Reference set
Pick clips that represent production: accents, crosstalk, phone-quality lines, jargon,
background noise. Have humans transcribe verbatim (or correct a model draft carefully).
Store as {"id": "call-07", "reference": "...", "hypothesis": "..."} JSONL once you have
model output.
2. Normalize
ffmpeg -hide_banner -i input.m4a -ac 1 -ar 16000 -c:a pcm_s16le -af "loudnorm" out.wav- For stereo call recordings with one speaker per channel, split channels instead
(
-af "pan=mono|c0=c0"andc0=c1) and transcribe each: free, perfect diarization. - Avoid aggressive noise reduction; it often hurts recognition more than the noise.
3-4. Transcribe with faster-whisper
from faster_whisper import WhisperModel, BatchedInferencePipeline
model = WhisperModel("large-v3", device="cuda", compute_type="float16") # cpu: "int8"
batched = BatchedInferencePipeline(model=model)
segments, info = batched.transcribe(
"out.wav",
batch_size=16,
language="en", # set when known; auto-detect misfires on short clips
vad_filter=True,
word_timestamps=True,
initial_prompt="Acme, Kubernetes, PgBouncer, SOC 2", # domain vocabulary and style
)
for seg in segments: # generator: transcription runs while you iterate
print(f"[{seg.start:7.2f} -> {seg.end:7.2f}] {seg.text}")| Setting | Guidance |
|---|---|
compute_type | GPU float16 (or int8_float16 to save memory); CPU int8 |
language | Always set it if known; detection on the first 30 s can be wrong |
vad_filter | On; tune min_silence_duration_ms if words are clipped |
initial_prompt | Names, acronyms, and spelling style; keep it short; it biases but does not force |
beam_size | 5 is the usual default; 1 is faster with a small accuracy loss |
condition_on_previous_text | Set False if one hallucination keeps repeating through the file |
Long files: VAD-based segmentation (as in the batched pipeline or WhisperX) beats fixed 30 s windows, which cut words in half. If you chunk manually, cut on silences and overlap 1-2 s, then deduplicate words in the overlap by timestamp.
5. Alignment and diarization
Whisper’s native timestamps drift on long files. For captions or speaker attribution, use forced alignment (WhisperX’s wav2vec2 alignment) to get word timings, then diarize:
from pyannote.audio import Pipeline
import torch
dia = Pipeline.from_pretrained("pyannote/speaker-diarization-community-1",
token=HF_TOKEN) # gated model: accept terms on the Hub
dia.to(torch.device("cuda"))
out = dia("out.wav", num_speakers=2) # pass the count when you know itAssign each word to the speaker turn that overlaps it most; smooth single-word speaker flips. Check the current pyannote release notes for the output object and argument names, which changed between major versions. Hosted APIs often include diarization; compare their speaker attribution on your audio too.
| Diarization failure | Fix |
|---|---|
| Too many speakers | Pass num_speakers or min_speakers/max_speakers |
| Speakers swap mid-call | Longer audio context; per-channel transcription if stereo |
| Short backchannels (“yeah”) misattributed | Merge turns shorter than ~0.5 s into neighbors |
6. Post-process
- Domain vocabulary: a replacement map for systematic misspellings (“pg bouncer” -> “PgBouncer”), applied with word boundaries.
- Optional LLM cleanup pass for readability (punctuation, paragraphing, removing filler) must preserve meaning; keep the verbatim transcript as the source of truth and never let the cleanup invent words. Diff a sample.
- PII redaction (card numbers, phone numbers) before storage if required.
7. Measure
python3 scripts/wer.py --jsonl pairs.jsonl --max-wer 0.15
python3 scripts/wer.py --ref reference.txt --hyp hypothesis.txtThe script normalizes both sides (case, punctuation, whitespace, optional number
words), computes WER and CER with substitution, deletion, and insertion counts, lists
the worst items and most frequent substitutions, and exits 1 if aggregate WER is above
--max-wer. Read the worst segments; they tell you whether the fix is vocabulary,
VAD, model size, or audio quality.
| Pattern in errors | Likely cause |
|---|---|
| Many insertions in quiet sections | Hallucination on silence; VAD off or too lenient |
| Deletions at chunk boundaries | Fixed-window chunking; switch to VAD segmentation |
| Same jargon wrong everywhere | Vocabulary; add to prompt and replacement map |
| High WER on one speaker or accent | Model size; test larger model or a vendor for that slice |
Typical targets: clean dictation under 5% WER, meetings 10-15%, phone calls 15-25%. Set your bar from what the downstream use tolerates.
8. Output
- SRT/VTT for captions: lines of at most ~42 characters, 1-2 lines per cue, cues of 1-7 s, split on punctuation and pauses. Use word timestamps to build cues.
- JSON with segments, words, speakers, and confidence for downstream processing.
- Plain text with speaker labels for human reading.
Gotchas
- Whisper translates instead of transcribing if
task="translate"slips in, and may answer in English on non-English audio when language detection fails. - Repeated-phrase loops on long audio: disable conditioning on previous text or segment with VAD.
- Timestamps from different tools use different conventions (segment vs word, start padding). Validate alignment on a few clips by ear.
- WER comparisons are meaningless unless both sides use identical normalization.
- Phone audio is 8 kHz; upsampling to 16 kHz is required but does not restore bandwidth. Expect higher WER and judge vendors on real phone audio.
- pyannote models are gated on the Hugging Face Hub; CI fails without a token and accepted terms.
Output contract
**Engine:** <model, compute type> or <vendor>; VAD <on/settings>
**Reference set:** <minutes>, <conditions>
**WER/CER:** <x>% / <y>% (normalization: <rules>); worst slice <name: x%>
**Diarization:** <method>, speaker attribution checked on <n> clips
**Outputs:** <SRT/VTT/JSON/text>, retention <policy>
**Throughput:** <x> min audio per min on <hardware>, cost <$/hour of audio>Bundled files
Loaded only when a step in the skill needs them. Review them here before installing.
scripts/wer.py
#!/usr/bin/env python3
"""Word and character error rate for transcripts, with consistent normalization.
Usage:
python3 wer.py --ref reference.txt --hyp hypothesis.txt [options]
python3 wer.py --jsonl pairs.jsonl [options]
pairs.jsonl lines: {"id": "call-1", "reference": "...", "hypothesis": "..."}
Options:
--max-wer 0.15 exit 1 if aggregate WER is above this
--keep-case do not lowercase
--keep-punct do not strip punctuation
--numbers map digits 0-20 and common number words to one form ("2" -> "two")
--worst 5 how many worst items to list
--top-subs 10 how many frequent substitutions to list
Aggregate WER = total (S + D + I) / total reference words, not the mean of per-item WERs.
Exits 2 on input errors.
"""
import argparse
import json
import re
import sys
import unicodedata
from collections import Counter
NUMS = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten",
"eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen",
"eighteen", "nineteen", "twenty"]
DIGIT_TO_WORD = {str(i): w for i, w in enumerate(NUMS)}
def normalize(text, keep_case=False, keep_punct=False, numbers=False):
text = unicodedata.normalize("NFKC", text)
if not keep_case:
text = text.casefold()
if not keep_punct:
text = text.replace("'", "'")
text = re.sub(r"(?<=\w)'(?=\w)", "", text) # don't -> dont
text = re.sub(r"[^\w\s]", " ", text)
words = text.split()
if numbers:
words = [DIGIT_TO_WORD.get(w, w) for w in words]
return words
def align(ref, hyp):
"""Levenshtein alignment; returns (S, D, I, substitution pairs)."""
n, m = len(ref), len(hyp)
d = [[0] * (m + 1) for _ in range(n + 1)]
for i in range(n + 1):
d[i][0] = i
for j in range(m + 1):
d[0][j] = j
for i in range(1, n + 1):
ri = ref[i - 1]
row, prev = d[i], d[i - 1]
for j in range(1, m + 1):
cost = 0 if ri == hyp[j - 1] else 1
row[j] = min(prev[j] + 1, row[j - 1] + 1, prev[j - 1] + cost)
s = dl = ins = 0
subs = []
i, j = n, m
while i > 0 or j > 0:
# prefer exact matches, then insertions/deletions, then substitutions on ties,
# so reported substitution pairs line up with what a human would mark
if i > 0 and j > 0 and ref[i - 1] == hyp[j - 1] and d[i][j] == d[i - 1][j - 1]:
i, j = i - 1, j - 1
elif j > 0 and d[i][j] == d[i][j - 1] + 1:
ins += 1
j -= 1
elif i > 0 and d[i][j] == d[i - 1][j] + 1:
dl += 1
i -= 1
else:
s += 1
subs.append((ref[i - 1], hyp[j - 1]))
i, j = i - 1, j - 1
return s, dl, ins, subs
def load_pairs(args):
if args.jsonl:
pairs = []
with open(args.jsonl, encoding="utf-8") as fh:
for n, line in enumerate(fh, 1):
if not line.strip():
continue
try:
o = json.loads(line)
pairs.append((str(o.get("id", n)), o["reference"], o["hypothesis"]))
except (json.JSONDecodeError, KeyError) as exc:
print(f"error: {args.jsonl}:{n}: {exc}", file=sys.stderr)
sys.exit(2)
return pairs
if args.ref and args.hyp:
with open(args.ref, encoding="utf-8") as r, open(args.hyp, encoding="utf-8") as h:
return [("file", r.read(), h.read())]
print("error: pass --jsonl, or both --ref and --hyp", file=sys.stderr)
sys.exit(2)
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--ref")
ap.add_argument("--hyp")
ap.add_argument("--jsonl")
ap.add_argument("--max-wer", type=float)
ap.add_argument("--keep-case", action="store_true")
ap.add_argument("--keep-punct", action="store_true")
ap.add_argument("--numbers", action="store_true")
ap.add_argument("--worst", type=int, default=5)
ap.add_argument("--top-subs", type=int, default=10)
args = ap.parse_args()
pairs = load_pairs(args)
tot = Counter()
sub_counts = Counter()
items = []
for pid, ref_t, hyp_t in pairs:
ref = normalize(ref_t, args.keep_case, args.keep_punct, args.numbers)
hyp = normalize(hyp_t, args.keep_case, args.keep_punct, args.numbers)
s, dl, ins, subs = align(ref, hyp)
sub_counts.update(subs)
rc, hc = list("".join(ref)), list("".join(hyp))
cs, cd, ci, _ = align(rc, hc)
tot.update(S=s, D=dl, I=ins, N=len(ref), CE=cs + cd + ci, CN=len(rc))
wer = (s + dl + ins) / len(ref) if ref else float(bool(hyp))
items.append((wer, pid, s, dl, ins, len(ref)))
if tot["N"] == 0:
print("error: references are empty after normalization", file=sys.stderr)
sys.exit(2)
wer = (tot["S"] + tot["D"] + tot["I"]) / tot["N"]
cer = tot["CE"] / tot["CN"] if tot["CN"] else 0.0
print(f"items: {len(pairs)} reference words: {tot['N']}")
print(f"WER: {wer:.2%} (S={tot['S']} D={tot['D']} I={tot['I']})")
print(f"CER: {cer:.2%}")
if len(items) > 1:
print(f"\nworst {min(args.worst, len(items))} items:")
for w, pid, s, dl, ins, n in sorted(items, reverse=True)[: args.worst]:
print(f" {pid:20} WER {w:6.1%} S={s} D={dl} I={ins} words={n}")
if sub_counts:
print(f"\nmost frequent substitutions (reference -> hypothesis):")
for (r, h), c in sub_counts.most_common(args.top_subs):
print(f" {r!r} -> {h!r}: {c}")
if args.max_wer is not None:
if wer > args.max_wer:
print(f"\nFAIL: WER {wer:.2%} above {args.max_wer:.2%}")
sys.exit(1)
print(f"\nPASS: WER at or below {args.max_wer:.2%}")
if __name__ == "__main__":
main()
About this skill
Written and maintained by the Claude Skills editors to the same editorial standard as every skill in the directory: a selective trigger, a real workflow, known pitfalls and a verification step. First published , last updated . MIT licensed.