Realtime Voice Agents skill for Claude Code

A voice agent is ready when voice-to-voice latency, measured from the caller’s side, is within budget at p50 and p95; it never answers before the caller has finished but never leaves dead air; it stops talking within a few hundred milliseconds when interrupted and forgets the words the caller never heard; slow tools are covered by speech; and a suite of recorded calls proves all of that on every change. A demo over office Wi-Fi proves none of it.

Ground rules

• Stream every stage. Nothing waits for a complete transcript, a complete LLM answer, or a complete audio file. Sequential batch stages cannot meet a conversational budget.
• Measure voice-to-voice: from the caller’s last voiced frame to the first agent audio the caller hears. Per-vendor TTFT numbers do not add up to that.
• Conversation history holds what was actually spoken, not what was generated.
• Write for the ear: one or two short sentences per turn, no markdown, lists or URLs. Numbers, dates and codes must be normalized before TTS (see text-to-speech).
• Confirm consequential actions (payments, cancellations, bookings) by reading back the key values and getting an explicit “yes”.

Choose the architecture

Option	Pick when	Costs
Cascaded: streaming STT -> LLM -> streaming TTS	You need a specific LLM, full text transcripts for compliance, per-stage vendor choice, or strict tool-calling control	More moving parts; latency is the sum of streamed stages
Speech-to-speech realtime API (OpenAI Realtime, Gemini Live)	Lowest latency and natural prosody matter most, and the provider’s model and tools fit	Less control over the reasoning model and voice; transcripts are secondary outputs; audio tokens priced separately
Hybrid: realtime model for conversation, text LLM behind a tool for hard reasoning	Most turns are simple, a few need a stronger model	Two models to evaluate

Default: build on an orchestration framework (Pipecat or LiveKit Agents) rather than raw sockets. Both handle transport, VAD, interruption plumbing, and support cascaded and speech-to-speech modes, so switching later is a config change rather than a rewrite. Check the framework’s current docs for parameter names; they change between releases.

Workflow

- [ ] 1. Write the call flows and the latency budget
- [ ] 2. Pick architecture, transport and region
- [ ] 3. Configure turn detection
- [ ] 4. Implement barge-in and history truncation
- [ ] 5. Add tools with spoken filler and timeouts
- [ ] 6. Build the recorded-audio test suite
- [ ] 7. Measure per stage, optimize the largest stage first
- [ ] 8. Production: call recording, latency dashboards, failure transfer

1. Latency budget

Human turn gaps are short, so treat these as planning targets and tune with real calls:

Stage	Target (p50)	Notes
Endpointing (speech end -> turn committed)	200-500 ms	Largest lever; fixed silence timers tend to sit at the top of this range
LLM time to first token	200-500 ms	Small or fast model, short prompt, cached prefix, no up-front reasoning
TTS time to first audio	100-250 ms	Streaming TTS fed sentence by sentence
Transport and buffering	50-150 ms	Jitter buffers, telephony legs, region distance
Voice-to-voice	under ~800 ms p50, under ~1.5 s p95	Above about 1.5 s callers start talking again or assume the line dropped

Every tool call adds its full duration on top unless it is covered with speech (step 5).

2. Transport and region

Client	Transport	Notes
Browser or mobile app	WebRTC	Opus codec, built-in echo cancellation, jitter buffer, NAT traversal
Phone (PSTN)	SIP trunk from a carrier or CPaaS, or the CPaaS media-stream WebSocket	Narrowband G.711 (mu-law/A-law, 8 kHz); resample to what STT expects and back for TTS
Server to server	WebSocket	Fine inside a data center; avoid over lossy networks

• OpenAI’s Realtime API accepts WebRTC, WebSocket and SIP; inbound SIP calls are pointed at sip.api.openai.com and accepted from a realtime.call.incoming webhook. Confirm the current model ID on the provider’s models page before shipping.
• Run the agent in the same region as the STT, LLM and TTS endpoints and the telephony media edge. Cross-ocean hops add latency to every turn.
• Support DTMF for digits and menus, and implement warm transfer to a human (SIP REFER or the CPaaS transfer API) as the failure path.

3. Turn detection

Method	Behavior	Use
VAD + silence timer (e.g. Silero VAD, server VAD)	Ends the turn after N ms of silence	Baseline; simple and predictable
Semantic turn detection (OpenAI semantic_vad, LiveKit turn detector model, STT endpointing such as Deepgram Flux or AssemblyAI)	Uses words and prosody to decide the caller is done	Default for open conversation; fewer cut-offs at the same speed
Push-to-talk / manual commit	Client commits the turn	Noisy environments, kiosks, walkie-talkie UX

OpenAI Realtime session config (GA shape; turn detection lives under audio.input):

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": {
        "turn_detection": { "type": "server_vad", "threshold": 0.5,
                            "prefix_padding_ms": 300, "silence_duration_ms": 500 }
      }
    }
  }
}

Swap to {"type": "semantic_vad", "eagerness": "auto"} for semantic detection; set turn_detection to null to commit turns manually.

Tuning rules:

• Callers cut off mid-sentence -> raise silence duration or lower eagerness; switch to semantic detection.
• Dead air after the caller finishes -> lower silence duration; check that STT finals are not the bottleneck.
• Collecting digits, addresses or card numbers -> lengthen the silence window for that step only (people pause between groups), or use DTMF.
• Background noise triggers turns -> raise the VAD threshold, enable noise suppression on the client, never on already-compressed telephony audio twice.

4. Barge-in

When the caller starts speaking while the agent is talking:

1. Stop playback immediately: flush the client or telephony output buffer, not just the generator. Audio already queued keeps playing otherwise.
2. Cancel in-flight LLM generation and TTS synthesis.
3. Truncate the assistant turn in history to what was played. Over WebSocket with OpenAI Realtime, send conversation.item.truncate with item_id, content_index and audio_end_ms; over WebRTC and SIP the server truncates unplayed audio itself. In a cascaded pipeline, track played duration and cut the text at the matching word.
4. Ignore false interruptions: backchannels (“mm-hmm”, “right”), coughs and echo. Require a minimum speech duration or word count before cancelling, and resume if the caller goes silent.

Echo cancellation is mandatory on speakerphone and in-browser use. Without it the agent hears itself and interrupts its own sentences. WebRTC provides it; raw WebSocket audio from a browser does not unless you enable it in getUserMedia constraints.

5. Tools mid-conversation

• Before any tool that may take longer than about 500 ms, speak a short filler (“Let me pull up your booking.”) in parallel with the call.
• Give every tool a timeout well under the caller’s patience (a few seconds), then say what happened and offer a next step instead of going silent.
• OpenAI Realtime flow: arguments stream in response.function_call_arguments.delta, the complete call is in response.done; reply with conversation.item.create of type function_call_output using the same call_id, then send response.create.
• Keep the tool list small and descriptions short; every tool definition adds tokens to each turn’s prompt and therefore to TTFT.
• Make write tools idempotent (a retried “book it” must not double-book) and gate them on a spoken confirmation.

6. Test with recorded audio

Build a corpus before tuning:

tests/audio/
  happy_path_booking.wav       # clean speech, expected outcome in a sidecar .json
  accent_variety_*.wav
  noisy_street.wav             # SNR well below office conditions
  long_pause_mid_sentence.wav  # "My account number is ... 4 4 7 ... 2 1"
  interrupt_during_answer.wav  # caller talks over the agent at a known offset
  backchannel_mmhmm.wav        # must NOT stop the agent
  silence_10s.wav              # agent should re-prompt, then end politely

• Stream files into the pipeline at real-time pace in 20 ms frames. Feeding faster than real time breaks VAD timing and hides latency.
• Include 8 kHz mu-law versions of each file if the agent answers phones.
• For multi-turn coverage, drive the agent with a simulated caller (an LLM persona plus TTS) and grade the transcript for task completion and policy compliance with llm-evals.
• Assert on outcomes (booking created, transfer triggered) and on latency per turn.

7. Measure and optimize

Log one record per agent turn with timestamps on one clock, then run:

python3 scripts/latency_report.py turns.jsonl --v2v-p50 800 --v2v-p95 1500 --barge-in-p95 300

It prints p50/p90/p95 for endpointing, LLM TTFT, TTS first audio, transport and voice-to-voice, plus tool and barge-in times, names the largest median stage, lists the slowest turns, and exits 1 if a budget is breached. Server timestamps miss the caller’s network leg: once per release, verify against stereo call recordings (caller and agent on separate channels) by measuring the gap in an audio editor or with a VAD script.

Optimization order: endpointing, then LLM TTFT (smaller model, shorter system prompt, prompt caching, no up-front reasoning on simple turns), then TTS (streaming, sentence-level chunking, warm connections), then region and transport.

Gotchas

• Sending the LLM’s first sentence to TTS only after the full response finishes; stream sentence by sentence.
• Opening a new TTS or STT connection per turn; keep connections warm for the call.
• Text formatting reaching TTS: asterisks, bullet points and URLs are read aloud.
• Clipping the first syllable because VAD prefix padding is too short.
• Telephony audio resampled with naive decimation, producing artifacts that hurt STT.
• History that keeps the full generated text after an interruption, so the agent later refers to things the caller never heard.
• Recording calls without the consent notice your jurisdiction requires; announce it.
• Testing only in quiet rooms with fast networks; production callers use speakerphones in cars.

Output contract

## Architecture: cascaded | speech-to-speech | hybrid (why), framework, transport, region
## Latency budget vs measured (scripts/latency_report.py: p50/p95 per stage)
## Turn detection settings and rationale
## Barge-in: stop path, truncation method, false-interruption rule
## Tools: list, filler phrases, timeouts, confirmations
## Recorded-audio suite: cases, pass/fail, outcome assertions
## Escalation: transfer path and triggers