📌 The claim, and why it holds
Hugging Face's speech-to-speech is a four-stage voice pipeline — VAD → STT → LLM → TTS — behind an OpenAI Realtime-compatible WebSocket API. Every stage is a swappable flag.
Verified against the repository: huggingface/speech-to-speech, 10,161 stars, 1,245 forks, Apache-2.0, last pushed 31 July 2026 — the same day the video went up. The README describes it in exactly those terms: "A low-latency, fully modular voice-agent pipeline: VAD → STT → LLM → TTS, exposed through an OpenAI Realtime-compatible WebSocket API. Every component is swappable."
What makes this video worth the seven minutes is not the feature list — it is that the presenter asks one question at the start and answers it with arithmetic at the end: "Which of those four stages is making the person wait?" Almost nothing in this genre poses a question that could come back with an inconvenient answer.
🔌 One line, not one rewrite 0:18
"You point the official OpenAI Python client at your own laptop. Same client code, same events. The one line you change is the base URL. It now reads 127.0.0.1."
This is the design decision everything else depends on, and it is correctly reported. The pipeline implements the OpenAI Realtime protocol, so an existing application does not learn a new API — it changes an endpoint.
Confirmed in the README, which uses
127.0.0.1 in four places and base_url in thirteen. The server starts at ws://localhost:8765/v1/realtime, and the documented pattern points the LLM slot at http://127.0.0.1:8080/v1 for a local llama.cpp backend.Why protocol compatibility beats feature parity. The expensive part of leaving a hosted API is rarely the model — it is the integration surface: event shapes, streaming semantics, error handling, the client library your code is built around. Reimplementing the wire protocol means none of that has to move. It is the same reason OpenAI-compatible endpoints won as a convention for text: compatibility is a distribution strategy, not a technical detail.
🎚️ VAD: the personality dial 1:17
Silero VAD v5 decides when you started and stopped speaking. The video's framing of it is the sharpest observation in the piece:
"That sounds like plumbing. It is the single biggest reason a voice agent feels polite or feels rude… Raise it, and the assistant waits while you think. Drop it, and it jumps on your pauses. That one number is most of its personality."
The 64 ms default checks out. The README documents
--min_silence_ms: "minimum length of silence intervals for segmenting speech. Default is 64 ms." It also exposes --min_speech_ms 384 and --min_speech_continuation_ms 192 as the recommended pairing, plus a merge window and a reopen cap — considerably more turn-taking control than the video has time to mention.Interruption is the part hosted APIs make hard. Switch on interrupt handling and talking over the assistant cancels mid-sentence. The video's aside — "try that with a request-response API" — is the real point: barge-in is a property of the transport, not the model. The README confirms the asymmetry from the other direction, noting that the minimal TCP socket mode "does not provide the full Realtime API feature set, including interruption handling".
👂 Transcription without Whisper 1:55
The default STT is NVIDIA Parakeet TDT 0.6B v3, not Whisper. The video gives three numbers; two are verifiable in NVIDIA's model card and both are right.
| Claim | Verified |
|---|---|
| 600M parameters, v3 | Card: "a 600-million-parameter multilingual ASR model" |
| 25 European languages, auto-detected | Card: "expanding language support from English to 25 European languages… automatically detects the language… without requiring additional prompting" |
| 6.34% average WER on the OpenASR leaderboard | Card benchmark table: parakeet-tdt-0.6b-v3 → 6.34% |
| ~3,000× real time | Not stated in the card in that form; RTFx figures live on the leaderboard rather than the card |
The framing worth keeping: "Same pipeline, different ears." Whisper is still available through Transformers or Faster Whisper, Lightning Whisper on MLX for macOS, and other backends for other languages — all as pip extras (
Note the licence difference the video does not mention: Parakeet is CC-BY-4.0, not Apache-2.0 like the pipeline around it. It has 747,858 downloads on v2 and 207,975 on v3, so this is a mainstream choice — but "the whole stack is Apache" is not quite right once you include the default weights.
[faster-whisper], [whisper-mlx]). The default changed; the option did not disappear.Note the licence difference the video does not mention: Parakeet is CC-BY-4.0, not Apache-2.0 like the pipeline around it. It has 747,858 downloads on v2 and 207,975 on v3, so this is a mainstream choice — but "the whole stack is Apache" is not quite right once you include the default weights.
🧠 The LLM slot is a socket 2:35
"This is where the repo flatly refuses to choose for you… The stage is a socket. What you push into it is your business."
Any OpenAI-compatible endpoint works: OpenAI itself, HF Inference Providers, OpenRouter, a vLLM server, a llama.cpp server — or skip the server entirely and run in-process through Transformers on a GPU or MLX-LM on Apple Silicon.
Confirmed: the README says the LLM slot "speaks OpenAI-compatible protocols, so you can point it at a hosted provider, at HF Inference Providers, or at a vLLM or llama.cpp server on your own hardware for a fully local, fully open stack."
This is the architectural claim that makes the rest measurable. If the model is a socket, you can swap it and time the difference — which is exactly what §6 does. A pipeline that hardcodes its model can tell you it is fast; one that does not lets you find out where it is slow.
🗣️ The voice, and first-packet time 3:03
Default TTS is Qwen3-TTS — 1.7B, voice cloning from about three seconds of reference audio. The video's numbers: streams in ~320 ms packets, first packet lands under 120 ms.
The observation attached to those numbers is the useful part: "That number matters more than the total, because the moment the first syllable arrives, the person stops feeling ignored."
That is the correct metric for conversational systems and it is routinely reported wrong. Total synthesis time is an engineering number; time to first audio is the number the human experiences. A system that starts speaking in 120 ms and finishes in two seconds feels responsive; one that is silent for 800 ms and then delivers instantly feels broken.
That is the correct metric for conversational systems and it is routinely reported wrong. Total synthesis time is an engineering number; time to first audio is the number the human experiences. A system that starts speaking in 120 ms and finishes in two seconds feels responsive; one that is silent for 800 ms and then delivers instantly feels broken.
Neither latency figure is in the README, and no benchmark is cited for them. The alternatives are all real and documented as pip extras — Kokoro-82M, Pocket TTS, ChatTTS, Meta MMS — but the 320 ms / 120 ms pair should be treated as the presenter's measurement or a vendor claim, not as repository-documented behaviour.
🎯 Naming the bottleneck 3:43
The payoff, and the reason this video is worth covering at all. Having built the four stages, the presenter adds them up:
| Stage | Contribution to perceived delay |
|---|---|
| VAD (Silero) | Milliseconds |
| STT (Parakeet) | "Effectively milliseconds" |
| TTS (Qwen3) | First audio inside a tenth of a second |
| LLM | Everything else |
"Three of the four stages are close to free, which leaves the language model holding the entire delay on its own. In a cascade, nothing can be spoken until the model has produced something to say. And it wants to think first."
Why this is a genuinely useful piece of analysis. Most voice-agent content optimises the wrong stage — better ASR, faster synthesis, a smaller VAD window — because those are the parts that feel like engineering. The arithmetic says they are already free. If your voice agent feels slow, the model is thinking, and nothing you do to the audio stages will fix it.
Note the structural constraint, which is a property of cascades rather than of this implementation: the TTS cannot start until the LLM emits its first usable text. That serialisation is the price of modularity, and it is what end-to-end audio models exist to remove.
Note the structural constraint, which is a property of cascades rather than of this implementation: the TTS cannot start until the LLM emits its first usable text. That serialisation is the price of modularity, and it is what end-to-end audio models exist to remove.
The Cerebras numbers need care. The video cites 1,851 output tokens/sec for Gemma 4 31B on Cerebras wafer-scale hardware, "around 35× a typical GPU endpoint", first token in 1.5 s, and a comparison placing Gemma 4 near Claude Haiku 4.5 on the Artificial Analysis Intelligence Index while running ~18× faster.
None of those figures appear in the README — I checked for each. What the README does contain is the exact invocation:
None of those figures appear in the README — I checked for each. What the README does contain is the exact invocation:
--model_name "google/gemma-4-31B-it:cerebras" pointed at https://router.huggingface.co/v1, with reasoning disabled "for low voice latency". So the configuration is real and documented; the throughput numbers are attributed to Artificial Analysis and were not independently reproduced here. Treat them as third-party benchmark claims.📊 The tail, not the average 4:58
"They are hunting the 95th percentile, not the average. It is the occasional 3-second stall that makes an assistant feel broken. Your median was fine. Your tail was the problem."
Twenty seconds, and the most transferable idea in the video. Latency in conversational systems is not experienced as an average — it is experienced as the worst pause the user remembers. A system with a 400 ms median and an occasional three-second stall is judged on the stall.
This generalises well past voice. Any interactive system where a human is waiting is graded on its tail, and reporting medians hides exactly the failures that lose users. It is the same reason a fluent-but-occasionally-wrong answer is worse than a consistently cautious one: people remember the outlier, not the distribution.
This generalises well past voice. Any interactive system where a human is waiting is graded on its tail, and reporting medians hides exactly the failures that lose users. It is the same reason a fluent-but-occasionally-wrong answer is worse than a consistently cautious one: people remember the outlier, not the distribution.
💸 Offline, and the cost trap 5:11
The reverse direction: nothing here requires a cloud. Run llama-server with a Gemma 4 GGUF, point the pipeline at 127.0.0.1:8080, done. On a Mac it collapses to a single optimal-settings flag putting all three models on Metal — Parakeet for ears, MLX-LM for thinking, Qwen3-TTS through mlx-audio for the mouth. "Works in airplane mode."
CPU-only is covered too: Kokoro-82M with 54 voices at roughly 6× real time without a GPU, or Pocket TTS at ~100M with first audio in about 200 ms.
The economics, checked
The pricing is exactly right. OpenAI's current pricing page lists
gpt-realtime-2.1 audio at $32.00 per 1M input tokens and $64.00 per 1M output — the two figures quoted, verbatim.The video then does the arithmetic most coverage skips: your speech is billed at one token per 100 ms, its speech at one per 50 ms. So a minute of you talking is 600 tokens; a minute of it answering is 1,200. Measured sessions land between roughly 5¢ and 46¢ per conversation minute.
One omission worth flagging. The same pricing page lists
gpt-realtime-2.1-mini at $10 / $20 per million audio tokens — roughly a third of the headline rate. The cost argument for self-hosting survives that comparison, but a fair version would have named the cheaper tier rather than pricing only the flagship.The deployment number that makes this concrete: the README states the pipeline "runs in production as the conversation backend for thousands of Reachy Mini robots". The video says over 9,000 units. Either way this is not a weekend demo — it is shipping hardware, which is a stronger argument than any benchmark.
⚖️ What it costs you to own 6:35
The section that earns the video its credibility, and it is unprompted:
"I will be straight about what it costs you. This is a cascade. Not one end-to-end audio model. So you own four moving parts and four ways to fail, with no support contract behind any of them. And it moves fast. The 2024 lineup — Parler TTS, MeloTTS, Distil-Whisper — is gone from the README. Pin your versions."
The churn claim is verifiable and true. The repository was created 7 August 2024 and the default backends have turned over since. The video says this itself at 0:59 — "the name is older than the thing… this repo went up in August 2024 as a fun demo" — which is an unusual correction to volunteer when the framing is "just open-sourced".
Two self-corrections in seven minutes. The presenter walks back the novelty framing (the repo is two years old, rebuilt) and names the maintenance cost (four failure modes, no support, pin your versions). Neither serves the video's own hook. That is the behaviour that separates an architectural breakdown from an announcement.
And the closing advice is the right one: "install it, run the recipe, and time each of the four stages yourself on your own hardware. The entire argument for this repo is that you can finally measure which stage is slow." The product being sold here is measurability, not speed.
And the closing advice is the right one: "install it, run the recipe, and time each of the four stages yourself on your own hardware. The entire argument for this repo is that you can finally measure which stage is slow." The product being sold here is measurability, not speed.
What is still missing: no discussion of concurrency (all figures are single-session), no memory footprint for running four models at once, and no quality comparison against the hosted alternative it costs against. Cheaper is established; equivalent is assumed.
🏦 Reading this from a bank
| Property | Implication in a regulated setting |
|---|---|
| Fully local, airplane-mode capable | Customer voice never leaves the perimeter — the decisive argument for regulated telephony, where recordings are personal data by default |
| OpenAI-protocol compatible | An exit path from a hosted vendor that does not require rewriting the application — concentration-risk mitigation you can actually evidence |
| Apache-2.0 pipeline | Commercially usable, auditable, self-hostable |
| Default STT is CC-BY-4.0, not Apache | Mixed licensing inside one stack — attribution obligations differ per component and need checking before deployment |
| Four components, no support contract | Four upgrade paths, four CVE surfaces, four things to pin. Named by the presenter, and it is the real operating cost |
| Voice cloning from ~3 s of audio | A capability with obvious fraud implications in a bank. Whatever else you do, this argues for consent controls and synthetic-speech detection on the inbound side |
| Interruption handling | Barge-in is table stakes for real telephony, and it is transport-level here rather than a model feature |
The point that transfers beyond voice. The value of this architecture is not that it is faster — it is that each stage is separately measurable and separately replaceable. In an environment where you must be able to explain what a system did and demonstrate control over each component, a modular cascade you can instrument beats an end-to-end box you cannot, even when the box is better on paper.
The flip side is the honest one the video names: four parts is four things to maintain, and nobody is on the other end of the phone when one of them breaks.
The flip side is the honest one the video names: four parts is four things to maintain, and nobody is on the other end of the phone when one of them breaks.
🔍 Claims checked
| Claim | Result |
|---|---|
| Repo exists, Apache-2.0, four-stage swappable pipeline | Confirmed — 10,161 ⭐, 1,245 forks, README matches the description |
| OpenAI Realtime protocol; change the base URL to 127.0.0.1 | Confirmed — 127.0.0.1 ×4, base_url ×13 in the README |
| Silero VAD, 64 ms default silence | Confirmed — --min_silence_ms, "Default is 64 ms" |
| Parakeet TDT 0.6B v3, 25 European languages, auto-detect | Confirmed verbatim in NVIDIA's model card |
| 6.34% average WER | Confirmed in the card's benchmark table |
| ~3,000× real time | Not in that form on the card; leaderboard figure, not reproduced here |
| Qwen3-TTS: 320 ms packets, first packet under 120 ms | Not in the README. Presenter's or vendor's measurement — no benchmark cited |
| OpenAI Realtime audio: $32/1M in, $64/1M out | Exact. OpenAI pricing page, gpt-realtime-2.1 |
| …but a cheaper tier exists | Omitted. gpt-realtime-2.1-mini is $10/$20 — roughly a third |
| Gemma 4 31B on Cerebras at 1,851 tok/s, ~35× a GPU endpoint | Not in the README. Attributed to Artificial Analysis; the :cerebras invocation itself is documented |
| Runs in production on thousands of Reachy Mini robots | README: "thousands of Reachy Mini robots". Video says 9,000+ — same order |
| Repo is from August 2024, backends have churned | Created 7 Aug 2024. The video says this itself — a volunteered correction |
| Kokoro-82M, Pocket TTS, ChatTTS, MMS, Faster-Whisper as options | All present as documented pip extras |
| "The whole stack is Apache 2.0" | Imprecise. The pipeline is; the default STT weights are CC-BY-4.0 |
Method note. YouTube rate-limited caption access, so the transcript was produced locally with Whisper large-v3-turbo — 129 segments across the full 7:24. It mangled several product names, as machine transcription does: Qwen3-TTS became "QN3 TTS" and "Quin 3", llama.cpp was split as "a LLAMA. CPP server", Paraformer became "Periphermer", and Reachy Mini became "Reaching Minis". All names above were corrected against the repository and vendor model cards rather than trusted from audio. Figures were checked against the README, NVIDIA's Parakeet card, the GitHub API and OpenAI's pricing page directly. Verified 1 August 2026.
💡 Key takeaways
- Three of the four stages are effectively free. VAD, STT and TTS are milliseconds; the language model holds the entire perceived delay by itself.
- Which means most voice-agent optimisation targets the wrong stage. If it feels slow, the model is thinking — faster ASR will not save you.
- Protocol compatibility is the distribution strategy. Reimplementing OpenAI's Realtime API means an existing app changes a URL, not an architecture.
- One number is most of the assistant's personality. 64 ms default silence before your turn ends: raise it and it waits, drop it and it interrupts.
- Barge-in is a transport property, not a model feature — which is why request-response APIs cannot offer it.
- Time to first audio beats total synthesis time. The moment the first syllable lands, the human stops feeling ignored.
- Hunt the 95th percentile, not the average. The occasional three-second stall is what makes an assistant feel broken; your median was fine.
- $32/$64 per million audio tokens is real — verified on OpenAI's pricing page — but a mini tier at $10/$20 exists and went unmentioned.
- A minute of you is 600 tokens; a minute of it is 1,200. Sessions land at 5–46¢ per conversation minute, before you multiply by a support queue.
- This ships on hardware. Thousands of Reachy Mini robots run it in production — a better argument than any benchmark.
- Modularity's price is serialisation. Nothing can be spoken until the model produces text; that is the cascade tax an end-to-end model avoids.
- Four parts, four ways to fail, no support contract. Named by the presenter, unprompted, alongside "pin your versions" — the repo is two years old and its defaults have already turned over once.
🔗 Resources & links
- Run Local Real-Time Voice AI at 1,800+ Tokens/SecSource video — Cloud Codes, 7:30, 31 July 2026
- huggingface/speech-to-speech10,161 stars, Apache-2.0. The README is detailed and worth reading directly
- NVIDIA Parakeet TDT 0.6B v3Default STT — 25 European languages, 6.34% average WER, CC-BY-4.0
- Silero VADThe turn-taking layer — where the 64 ms personality dial lives
- Kokoro-82MThe CPU-only TTS option — 54 voices, ~6× real time without a GPU
- OpenAI pricingWhere the $32/$64 audio rates are confirmed — and the $10/$20 mini tier is listed
- Reachy MiniThe robot running this pipeline in production
🕐 Timestamp index
0:00The whole voice loop, open-sourced
0:13Four stages: VAD → STT → LLM → TTS
0:18"The one line you change is the base URL"
0:42"Which of those four stages is making the person wait?"
0:59Self-correction: the repo is from August 2024
1:17Silero VAD v5 — turn detection
1:32The 64 ms dial — "most of its personality"
1:44Interruption handling
1:55Parakeet TDT replaces Whisper as default
2:096.34% WER, ~3,000× real time
2:35The LLM slot — "the stage is a socket"
3:03Qwen3-TTS and voice cloning
3:24Why first-packet time beats total time
3:43The arithmetic — the LLM holds the delay
4:02The Cerebras recipe — Gemma 4 31B
4:181,851 tokens/sec, ~35× a GPU endpoint
4:47Switching it on is one flag
4:58"Your median was fine. Your tail was the problem."
5:11Fully local — llama.cpp and the Mac flag
5:41CPU-only: Kokoro and Pocket TTS
5:58The cost trap — $32/$64 per million
6:155–46¢ per conversation minute
6:229,000+ Reachy Mini robots in production
6:35"Four moving parts and four ways to fail"
6:54"Pin your versions"
7:05Go time the four stages yourself