1 Beyond "What Was Said"
Hervé Bredin, chief science officer and co-founder of pyannote.ai, opens with his background: academic researcher for most of his career, then founded pyannote.ai two years ago. He built the pyannote open-source toolkit for speaker diarization, which became hugely popular after OpenAI released Whisper — people needed to combine speech-to-text with speaker identification.
The talk builds a layered view of conversation understanding:
- Transcription — "what was said" (Whisper, Parakeet, etc.)
- Speaker-attributed transcription — "who said what"
- Temporal understanding — "who said what and when"
- Paralinguistic understanding — "who said what, when, and how"
Most current Voice AI stops at step 1 or 2. Bredin argues the real value is in steps 3 and 4.
2 Why "Who" Matters — Real Applications
Bredin gives concrete examples where knowing the speaker is as important as the words:
- Automatic video dubbing — translating a video to another language requires consistent voice assignment per speaker
- Meeting note-takers — assigning action points to the right person ("Alice said she'll handle the backend")
- Podcast intelligence — tracking a guest across different episodes or different podcasts
Evidence of demand: on Hugging Face's model repository, filtering by the "audio" tag and sorting by downloads, 3 of the top 7 models are related to speaker identity and diarization — the other 4 are speech-to-text.
3 "When" and "How" — Deeper Understanding
Precise timestamps unlock information that flat transcripts miss:
- Interruptions — detecting when one speaker cuts off another (signals disagreement, urgency)
- Back-channels — small "mm-hmm", "okay" sounds indicating agreement or engagement
- Pauses — hesitation between speech turns can reveal uncertainty or emotional state
And the "how" dimension adds even more:
- Laughter — was it humor or nervousness?
- Stress and prosody — "the DOG ate the cake" vs "the dog ate THE CAKE" have different meanings based on word emphasis
- Disfluency — stuttering or rephrasing reveals cognitive load
4 Speaker Diarization — The Pipeline
Diarization answers "who speaks when" and involves three stages:
- Voice Activity Detection (VAD) — is anyone speaking at this moment?
- Segmentation — split speech regions into individual turns, finding speaker change points and overlapping regions
- Speaker assignment — assign an identity label (speaker 1, speaker 2, etc.) to each segment
What makes it hard
- The number of speakers is unknown — can't use standard classification
- Speaker labels are permutation-invariant — swapping "green" and "black" labels is still correct
- Overlapping speech — two people talking simultaneously
- Very short speech turns — back-channels ("mm-hmm") that carry critical meaning
- Imbalanced speech time — one dominant speaker, several quiet ones
5 DER — Diarization Error Rate
DER is the standard evaluation metric, consisting of three error types:
| Error Type | What It Means |
|---|---|
| Confusion | Detected speech but assigned the wrong speaker |
| False Alarm | Detected speech when no one was actually speaking |
| Missed Detection | Failed to detect speech (common during overlapping segments) |
DER = (confusion + false alarm + missed detection) / total speech duration
6 Live Demo — Community vs. Precision Models
Bredin runs a live demo on a 30-second phone call between two women (Diane from New Jersey, Sheila from Texas). Using a Python notebook:
- pyannote Community 1 model (open source, runs on his Mac via MPS) → 5% DER
- pyannote Precision 2 model (proprietary, via cloud API) → 3% DER
The visualization shows color-coded speaker timelines with errors highlighted: confusion in one color, false alarms and missed detections in others. The Precision 2 model makes fewer boundary errors and handles overlapping speech better.
7 State of the Art — It Depends on the Use Case
When asked "how good is diarization today?" — it varies enormously:
| Scenario | Best DER |
|---|---|
| Clean telephone speech (2 speakers) | ~2% |
| Meeting room (headset mic) | ~8% |
| Noisy restaurant (multiple speakers) | ~41% |
8 The Reconciliation Problem
Combining diarization ("who speaks when") with transcription ("what was said") should be simple — just assign each word to a speaker. In practice, it's much harder:
- Timestamps disagree — STT and diarization produce different time boundaries
- Words fall between speakers — a word's timestamp lands in a gap between two speaker segments
- Overlapping speech — STT models trained on single-speaker data fail on multi-speaker audio
- Ghost detections — diarization detects speech that STT doesn't transcribe, and vice versa
9 Benchmark Reality Check — Headset vs. Table Mic
This is the bombshell slide. Nvidia's Parakeet model reports 11.4% word error rate on the AMI meeting dataset on the Open ASR Leaderboard. Bredin runs the exact same model on the exact same dataset and gets 26%.
The difference? The leaderboard uses headset microphone audio (clean, single-speaker per channel). Bredin uses the table microphone (distant, multi-speaker, with overlap and crosstalk). Same model, same meetings, completely different results.
10 Live Demo — Speaker-Attributed Transcription
Bredin demonstrates the word-level reconciliation problem live. Using Nvidia Parakeet for transcription + pyannote Precision 2 for diarization on the same phone call:
- Most words clearly belong to one speaker — easy assignment
- But the word "Oh" falls exactly between two speaker segments — which speaker owns it?
- Another word "Okay" has two speakers talking over it but only one transcribed word
pyannote.ai's proprietary STT Orchestration solves this by reconciling diarization and transcription, even interleaving words during overlapping speech. In the demo, overlapping speech at "in New Jersey" / "And I'm Sheila" is correctly separated word by word.
How it works (partially revealed)
The core technique is called exclusive diarization: during overlap, the system selects the most likely speaker that the STT model will successfully transcribe, simplifying the reconciliation. The full approach is proprietary, but this component is available in the open-source Community 1 model.
11 pyannote Ecosystem & Tooling
The full pyannote toolkit stack:
- pyannote — open-source diarization toolkit (~10K GitHub stars)
- pyannote.metrics — evaluation library (DER calculation)
- ipyannote — interactive visualization widget for Jupyter notebooks (the "i" stands for "interactive", not iOS)
- pyannote.ai SDK — access to premium models (Precision 2) via cloud API
- pyannote tutorials — GitHub repo with the demo notebook from this talk
🎯 Key Takeaways
🔑 Key Takeaways
- Transcription alone is not enough — "who said what" is as important as "what was said" for real conversation understanding
- Speaker diarization is unsolved for hard cases — 2% DER on clean phone calls, 41% in noisy restaurants
- Benchmarks lie by omission — the same model scores 11.4% (headset) vs 26% (table mic) on the same dataset
- Combining STT + diarization is harder than it looks — timestamps disagree, words fall between speakers, overlap breaks single-speaker STT models
- pyannote's "exclusive diarization" simplifies reconciliation by selecting the most transcribable speaker during overlap
- Temporal precision matters — interruptions, back-channels, and pauses carry meaning that flat transcripts lose
- Prosody is the next frontier — word stress, tone, and disfluency can change sentence meaning entirely
- pyannote Community 1 is open source and free — 5% DER on clean phone calls, runs locally on Mac (MPS)
- Precision 2 is proprietary via cloud API — 3% DER, better boundary detection and overlap handling
- 3 of the top 7 audio models on Hugging Face are diarization-related — showing massive community demand
🔗 Resources & Links
- pyannote-audio — open-source speaker diarization toolkit
- pyannote.ai — commercial platform with Precision 2 model
- Hervé Bredin on GitHub
- @hbredin on X