How Far Are We from "Her"?

1 From the Movie to GPT Live

The video opens with Spike Jonze's 2013 film Her, where Theodore (Joaquin Phoenix) falls in love with Samantha, an AI assistant who speaks, listens, understands emotion, and never misses a beat. Thirteen years later — July 2026 — that fiction is becoming engineering reality.

Two major announcements frame the discussion:

• GPT Live from OpenAI — a full-duplex voice mode that can speak and listen simultaneously
• Thinking Machines (Mira Murati's new lab) — announced "interaction models" that respond in real time and can proactively jump into conversation

Then there's Neil Zeghidour (Nao), who spent years at FAANG labs building the audio tech behind Google Meet and NotebookLM, then founded Kyutai — a Paris-based nonprofit that built the first full-duplex voice assistant.

2 ChatGPT Standard — The Cascade Architecture

ChatGPT's "Standard" voice mode is an LLM wrapper built on GPT-5.5 — a pure language model that cannot process audio natively. To give it a voice, OpenAI chains together a pipeline:

1. VAD (Voice Activity Detection) — binary decision every 30ms: is someone speaking?
2. ASR (Automatic Speech Recognition) — converts speech to text
3. LLM — processes the text and generates a response
4. TTS (Text-to-Speech) — converts the response back to audio

This is a cascade — latency accumulates across each component. Predecessors took multiple seconds to respond; humans take roughly 200ms in natural conversation.

The demo powerfully shows the limitations: Standard mode cannot detect whispering, cannot process beatboxing, and strips away all the richness of how something is said — only capturing what is said.

3 ChatGPT Advanced — End-to-End Models

GPT-4o introduced the omnimodal approach — the "o" stands for omni. Instead of chaining separate models, a single model processes text, audio, image, and video without a text bottleneck.

• The complete opposite of cascades — all components trained together on speech data
• Input acoustics are never lost: the model hears tone, emotion, cadence
• Latency approaches human-level (~200ms)

But there's a catch: GPT-4o is half-duplex. It can either speak or listen at any given moment, never both simultaneously.

4 ChatGPT Live & Full Duplex

Backchanneling is a core challenge: when a human says "mhm" or "yeah" mid-conversation, it signals attention — not interruption. Voice assistants must decide: should the user's interjection break the current response, or is it just acknowledgment?

GPT Live solves this by fundamentally rethinking how conversations work:

• Throws away the concept of speaker turns entirely
• Conversations are not turn-based — they're time-based
• Audio is cut into 20ms chunks, continuously streaming in and out
• This enables catching interruptions, simultaneous speech, and natural backchanneling

5 Moshi — The First Full Duplex Assistant

Moshi is the open-source full-duplex voice assistant from Kyutai, Neil Zeghidour's nonprofit lab in Paris. Built by a team of fewer than 10 people in under 6 months.

Moshi remained the only full-duplex model in the world for almost two years — until GPT Live launched in July 2026.

How it works at a high level:

• Takes two parallel audio streams: user's microphone input + its own generated response
• Handles backchanneling naturally — "mhm" doesn't force a turn switch
• Both streams are processed simultaneously and continuously

6 How Moshi Works — Audio Tokenization

To generate audio with a language model, you need to split it into discrete building blocks that can be predicted one at a time — just like text tokens.

The naive approach fails: raw audio amplitudes would require 24,000 sequential predictions per second. Far too many.

The solution: fixed-size frames of approximately 80ms — the audio equivalent of text tokens. But audio is richer than text. A single audio frame carries both:

• Semantics — what is being said (phonemes, words, meaning)
• Acoustics — how it's being said (pitch, tone, emotion, speaker identity)

The decomposition (from Neil Zeghidour's work at Google with AudioLM):

This decomposition is conceptually elegant: think first about what to say, then about how to say it — mirroring how humans produce speech.

7 The Inner Monologue

Text sits at the "top of the abstraction pyramid" — invented over 5,000 years ago, it's an incredibly efficient signal. Moshi removed the text bottleneck from cascades, but text is "pure upside" as long as audio also goes through.

The inner monologue is a text stream output alongside the audio stream. Crucially, this is not like chain-of-thought in LLMs — there's no delay for reasoning. The text advances at the same pace as the audio.

Architecture versatility: Moshi is an augmented LLM with a multimodal vocabulary (text + audio tokens). A big portion of weights are initialized from a pre-trained LLM, then fine-tuned on speech data. This single architecture yields three behaviors:

• Delay the text stream → speech-to-text model
• Delay the audio stream → text-to-speech model
• Run both → full speech-to-speech model with inner monologue

8 Why Aren't Full Duplex Models Everywhere?

Here's the surprising twist: Neil Zeghidour — who built Moshi — spun off Gradium, a company that builds cascade systems. Why would the creator of full duplex go back to cascades?

The end-to-end disadvantage: you can't easily swap in the newest LLM. Upgrading requires retraining the entire pipeline. With cascades, you just plug in a better LLM and everything else stays the same.

Unmute (also from Kyutai) is a streaming STT + streaming TTS framework that lets you take any LLM and give it a voice. It drew more interest from big companies than Moshi — because companies already had powerful text models and just wanted to add voice.

Architecture Comparison

Cascades are the practical approach today — think bank customer support. But that's the short-term view. OpenAI and Thinking Machines are investing in end-to-end for the long term.

9 Tool Calling & Moshi RAG

In Her, Samantha does tool calling — she accesses Theodore's Gmail to read his emails. Both OpenAI and Thinking Machines confirm they handle this: the core speech-to-speech model delegates to an async background LLM, keeping the conversation going while background processing happens.

Moshi RAG takes this further — pulling knowledge from external documents mid-conversation without pausing:

1. When Moshi needs outside knowledge, it drops a special token into the inner monologue
2. This fires a retrieval request to an external system
3. Meanwhile, Moshi fills the silence with a rough answer or small talk: "Let me check that for you" — just like a human customer support agent
4. The retrieved document is streamed as a fourth stream alongside the existing three (user audio, model audio, inner monologue text)

10 The Future — How Far Are We from Her?

Neil Zeghidour predicts end-to-end models will become dominant in 1–2 years ("being a bit pessimistic"). Two waves are coming after that:

• Miniaturization — models running on-device, no cloud required
• Robot integration — voice AI embedded in physical agents

The key challenge is post-training: creating the right synthetic in-situ data and finding the right reward models. The hard problem is fundamentally subjective:

The current status is "sort of close and sort of far" — the same paradox as AGI. The goalpost keeps moving. AI becomes superhuman in one dimension, then reveals itself as "super dumb" in another.

The video closes with an important ethical reflection:

• ✅ Better customer support — clearly beneficial
• 🤔 AI therapist — maybe beneficial, needs care
• ❌ AI friendship / love — problematic territory

📚 Resources & Links

• Moshi Paper (arXiv) The original research paper on the Moshi full-duplex voice assistant
• Moshi RAG Paper (arXiv) Retrieval-augmented generation for real-time speech models
• Thinking Machines — Interaction Models Mira Murati's lab announcing real-time, proactive interaction models
• Introducing GPT Live — OpenAI OpenAI's announcement of full-duplex voice mode
• Kyutai Paris-based nonprofit AI lab — creators of Moshi
• Gradium Neil Zeghidour's spinoff building cascade voice systems
• Unmute (Kyutai) Streaming STT + TTS framework — give any LLM a voice
• Audio Tokenization Explained — Julia Turc Deep dive into how audio is converted into tokens for language models
• Original Video — How Far Are We from "Her"? Julia Turc · 21:29

⏱ Timestamp Index