{ "video": { "id": "a8EcVumh71E", "title": "Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium", "duration": 1154, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "a8EcVumh71E", "title": "Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium", "one_liner": "Gradium's CEO traces voice agents from Siri to speech-to-speech models to argue that every shipping real-time model is still half-duplex — a walkie-talkie — and that the fix is not one giant speech-to-speech model but a small, on-device, full-duplex voice interface that delegates thinking to a background text LLM.", "summary": "Neil Zeghidour walks through four generations of voice interfaces — Siri's closed-ended STT→NLU→action pipeline (2011), OpenAI's original voice mode (open-ended but no agentic ability, multi-second latency), cascaded STT-LLM-TTS voice agents with tool calls, and speech-to-speech models like OpenAI's advanced voice mode — showing that each gain in naturalness has cost intelligence, because a text model's fixed capacity gets spent on handling audio. He argues the deeper unsolved problem is turn-taking: even lightning-fast speech-to-speech models are half-duplex, so backchanneling ('mhm, yeah') derails them, which he demonstrates live. He explains how audio tokenizers/neural codecs make audio LLMs tractable at all, and how multi-stream language models (Moshi, 2024) made the first genuinely full-duplex model. The talk closes on two futures — scale the speech-to-speech model (OpenAI's apparent path) versus split naturalness from intelligence (Gradium's Moushiraq hybrid) — and bets on the hybrid on cost and backend optionality grounds.", "key_points": [ "Four generations framed by three axes — open-endedness, agentic capability, naturalness, latency: Siri (2011) was a constrained closed-ended pipeline with no LLM, just natural language understanding classifying which app and action to trigger; OpenAI's original voice mode opened the dialogue but had no agentic capability and multi-second latency; cascaded voice agents restored tool calling, reasoning and planning with better latency; speech-to-speech collapsed STT/LLM/TTS into one model with near-ideal latency and the ability to read non-linguistic information.", "Every gain in naturalness has cost intelligence: cascaded agents are 'as smart as the best text models', while speech-to-speech models remain 'fundamentally still limited' — because an LLM has a fixed capacity in its weights, and teaching it to understand and produce speech takes that capacity away from intelligence.", "Turn-taking, not latency, is the remaining wall. Speech-to-speech is not the same as full duplex — GPT-realtime-style models are half duplex (listening or speaking), so a live backchanneling demo shows the model repeatedly stopping and apologising when the user just says 'yeah'. On a phone call with a relative, up to 20% of the time both people are speaking at once.", "Raw audio is untrainable for an LLM: an eight-word sentence takes ~3 seconds, which at 24 kHz is 72,000 timesteps instead of 8 tokens; with quadratic attention complexity a 10,000× longer sequence is 100 million times more expensive. Neural codecs (audio tokenizers) — encoder to a dense compressed representation, decoder back to high-quality audio — are what make audio LLMs possible.", "Full duplex comes from multi-stream language models: a transformer that models two token sequences at once so both parties can be active, inactive, or overlapping. Gradium/Kyutai shipped this first in Moshi (August 2024); Zeghidour says Thinking Machines now uses the same technique for its interaction model and OpenAI most likely does for its bidirectional model.", "The Moshi demo (a spaceship-trajectory conversation) shows the model guessing the end of the question and answering over the speaker without breaking flow; it was 'even a bit irritating' because it interrupted constantly, but stayed usable in extremely noisy environments with coughing and background noise.", "Path 1 — scale the speech-to-speech model with better pre/post-training — gives a single model to orchestrate and easy deployment, but the text→speech-to-speech conversion is 'extremely complex and costly': OpenAI's advanced voice mode was still powered by GPT-4o long after several new generations of text model shipped.", "Path 2 — Gradium's Moushiraq paper, also reused by Thinking Machines' interaction model — splits naturalness from intelligence: a small, possibly on-device, full-duplex speech-to-speech interface whose only job is to hold a natural conversation and 'know when it doesn't know', delegating reasoning, tool calls and planning asynchronously to one background text LLM serving hundreds to thousands of voice frontends.", "Two arguments for the hybrid: cost — 'it's a loss of money to do chit-chat with gigantic speech-to-speech models that can resolve differential equations' — and optionality, since a speech-to-speech model ties your hands to one provider while people want to swap the backend the day a new model drops. Cascaded systems are still most of the voice market today." ], "takeaways": [ "Stop treating latency as the naturalness metric. Test your voice agent against backchanneling and overlapping speech specifically — if the model stops every time a user says 'mhm', you have a half-duplex walkie-talkie no matter how fast it responds.", "Choose architecture by what the use case actually needs: cascaded STT-LLM-TTS if you need frontier intelligence and reliable tool calling; end-to-end speech-to-speech only where naturalness and latency dominate and the model can afford to be less smart.", "Don't route every turn through a giant multimodal mixture of experts — separate the conversational interface from the reasoning backend so chit-chat is cheap and only hard turns hit the expensive model.", "Weigh vendor lock-in explicitly: committing to a single-provider speech-to-speech model means you inherit their upgrade cadence, which lagged the text frontier by several model generations in the GPT-4o advanced-voice-mode case.", "If you need a full-duplex model, look at multi-stream language modelling (two simultaneous token streams) rather than trying to patch turn-taking with VAD tuning." ], "topics": [ "voice-agents", "speech-to-speech", "full-duplex", "audio-llms", "neural-codecs", "latency", "model-architecture", "on-device" ], "tools": [ "Gradium (gradium.ai)", "Kyutai", "Moshi", "Hibiki", "Moushiraq", "Siri", "OpenAI voice mode / advanced voice mode", "GPT-4o", "GPT-realtime", "Thinking Machines interaction model", "ElevenLabs", "Anthropic" ], "quotes": [ { "text": "you can have lightning speed latency. Fundamentally, this is uh an issue that can not be resolved when you're using turn-taking. So, here that's the walkie-talkie.", "at": "09:20", "url": "https://www.youtube.com/watch?v=a8EcVumh71E&t=560s" }, { "text": "If you're having a relative on the phone, there is up to 20% of the time where you are both speaking at the same time.", "at": "09:52", "url": "https://www.youtube.com/watch?v=a8EcVumh71E&t=592s" }, { "text": "fundamentally, there is a cost of adding a new modality to a text model that is going to be paid in in intelligence.", "at": "15:01", "url": "https://www.youtube.com/watch?v=a8EcVumh71E&t=901s" }, { "text": "it's a loss of money to do chit-chat with gigantic speech-to-speech models that can resolve differential equations and so on.", "at": "17:32", "url": "https://www.youtube.com/watch?v=a8EcVumh71E&t=1052s" } ], "words": 3880 }, "summary_url": "/#a8EcVumh71E", "transcript": { "html": "/transcripts/a8EcVumh71E.html", "txt": "/transcripts/a8EcVumh71E.txt", "vtt": "/transcripts/a8EcVumh71E.vtt" } }