Overview
Trelis Research presents Chorus Pro, a dedicated multi-speaker transcription model that outperforms Gemini Pro 3.1 on meetings with overlapping speakers — especially when there are 4+ participants. The video benchmarks both systems on the AMI and ICSI meeting datasets using the cpWER (concatenated permutation Word Error Rate) metric, walks through actual transcript comparisons showing Chorus Pro's overlap detection and timestamping advantages, and demonstrates Gemini's failure modes: collapsing to 4 speakers maximum, failing to return valid JSON on longer audio, and producing incorrect timestamps on multi-speaker content.
1 The Overlapping Speaker Problem
Meeting transcription remains a hard problem for two reasons:
- Overlapping speakers — multiple people talking simultaneously (backchanneling, interruptions, side conversations)
- Speaker identity across long audio — matching the same speaker consistently across tens of minutes or hours
General-purpose models like Gemini handle this as a side task; Chorus Pro is a dedicated, smaller model built specifically for multi-speaker transcription — and because it's purpose-built, it can compete with and often beat Gemini on this narrow application.
2 cpWER Metric Explained
Standard transcription uses WER (Word Error Rate) — percentage of incorrect words vs ground truth. For multi-speaker scenarios, the metric becomes cpWER:
- P = Permutation — find the permutation of speaker tracks that most closely matches the ground truth (since speaker labels are arbitrary)
- C = Concatenated — border the best-matched chunks together and score as one long sequence
The cpWER values in the benchmarks are notably high (bad) — reflecting how genuinely difficult this task is, even for state-of-the-art systems. Note: cpWER penalizes wrong words and wrong speaker attribution, but does not penalize timestamp errors.
3 Chorus Pro vs Gemini — Benchmark Results
Benchmarked on two standard meeting datasets:
AMI dataset (up to 4 speakers, <1 hour meetings)
- Performance is fairly similar between Chorus Pro and Gemini
- Chorus Pro scores appear slightly inflated because it correctly transcribes audio that's missing from the ground truth reference
ICSI dataset (up to 8 speakers, longer meetings)
- Chorus Pro significantly outperforms Gemini
- Gemini struggles once you go beyond 4 speakers — "it will typically just predict four speakers"
- Gemini is unreliable — sometimes fails to respond in correct JSON format, especially on longer audio
- When Gemini does succeed on 5+ speakers, it collapses timestamps into incorrect values
4 AMI Transcript Comparison
Side-by-side comparison of reference transcript, Gemini output, and Chorus Pro output on an AMI meeting:
- Chorus Pro appears to have some early mistakes, but it's actually correctly picking up audible audio that the reference ground truth missed entirely
- Both AMI and ICSI test sets have ground truth errors — segments with audible speech that aren't in the reference transcript
- Chorus Pro catches overlapping speech and marks it as such; Gemini transcribes the same segments as non-overlapping
The host acknowledges this inflates Chorus Pro's cpWER score and commits to making a fairer comparison by removing these reference gaps in a future video.
5 ICSI Results — Gemini's JSON & Speaker Failures
On ICSI meetings (5-8 speakers), Gemini shows critical failure modes:
- JSON format failures — when asked to transcribe with timestamps, Gemini often fails to return valid JSON, especially on longer audio
- 4-speaker ceiling — Gemini appears to have been trained on at most 4 speakers. It collapses 5-8 speakers into 4 tracks
- Timestamp collapse — even when scoring 66% cpWER (which doesn't penalize timestamps), Gemini produces incorrect timestamp values
Chorus Pro maintains performance consistently across the full meeting duration and handles 5-8 speakers correctly, though it occasionally attributes speech to phantom extra speakers (speaker 6 or 7). These phantom speakers have minimal text, so the cpWER impact is small.
6 Timestamp Accuracy & Overlap Detection
Chorus Pro's key differentiators beyond raw transcription accuracy:
- Accurate timestamps — including for overlapping segments. Gemini cannot timestamp multi-speaker audio reliably
- Overlap detection — backchannel filler ("yeah," "uh-huh") is correctly identified as overlapping with the main speech, with accurate time positioning
- Visual shape preservation — the transcript structure visually matches the reference, showing that speaker attribution and timing are consistently correct across long meetings
Export options include VTT and SRT subtitle formats, which reveal the precise timing of overlapping speech that's invisible in plain text view.
7 Live Demo & API Access
The model is available at router.trelis.com:
- Login: GitHub account (Google login coming soon)
- Upload any audio file and click transcribe
- API access available for programmatic use
- Export formats: plain text, VTT, SRT
- Language support: primarily English, other major languages in development
- Enterprise services: custom performance tuning for specific use cases
🎯 Key Takeaways
🔑 Key Takeaways
- Dedicated beats general-purpose for multi-speaker transcription — Chorus Pro (a smaller, focused model) outperforms Gemini Pro 3.1 on meetings with overlapping speakers
- Gemini has a 4-speaker ceiling — it collapses 5-8 speakers into 4 tracks, likely a training data limitation. Chorus Pro handles 8+ speakers correctly
- cpWER is the right metric — standard WER doesn't account for speaker attribution. cpWER finds the best speaker permutation match, then scores concatenated chunks
- Gemini's JSON output is unreliable on long audio — it fails to return valid structured output and produces incorrect timestamps for multi-speaker content
- Overlap detection is critical for accuracy — backchannels ("yeah," "uh-huh") need to be recognized as overlapping speech, not forced into a linear transcript
- Ground truth datasets have errors — both AMI and ICSI have missing speech segments, which can inflate scores for models that correctly transcribe those gaps
- Timestamps matter for downstream use — cpWER doesn't penalize timestamps, but real applications (meeting summaries, search, navigation) depend on accurate time alignment
- VTT/SRT export reveals overlap — plain text hides the temporal relationship between overlapping utterances; subtitle formats expose the true timing
- Meeting transcription is still hard — even state-of-the-art systems show high error rates, reflecting the genuine difficulty of overlapping multi-speaker audio
- Available now — Chorus Pro is accessible at router.trelis.com with both UI and API access, English-first with more languages coming
🔗 Resources & Links
- Chorus Pro — router.trelis.com — try the model, upload audio, export transcripts
- Trelis Voice AI Services — enterprise ASR, TTS, and agent services