Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium

AI Engineer · 19 min · 245 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Okay, hi everyone.
00:14I'm Nel co-founder and CEO of of Gradio.
00:18So Gradio is a startup based in Paris.
00:22Most of our background is from research.
00:24In particular, we have invented algorithms such as audio LLMs, speech speech-to-speech models, neural codex, and so on and so forth.
00:32And basically we started from a research project called QTI, a non-profit research lab that has been focusing on voice since day one.
00:41So in particular, we released in 2024 the first full duplex speech-to-speech model called Moshi, the first real-time speech-to-speech translation system called Hibiki, and the first TTS model that can run locally on a on a smartphone.
00:56And basically I will just say a few words about what we do, but we are a model company that trains models for building voice agents and voice applications.
01:04So we do TTS, API, and on-device speech-to-text, speech-to-speech translation, and much more to come.
01:11What we do is that we train foundation models for audio, and then we can apply them for a lot of different tasks.
01:17So I go quickly on voice agents because it's the fourth talk about the topic, but basically now we have these voice interfaces that we can use to do a lot of things across a variety of products and types of interactions with NPCs,
01:32with customer agents, language learners, coach, and so on and so forth.
01:37And in this talk I tried to go through the history of this technology and where I see it going in the next years.
01:46And maybe to start, I think we can take a look at the announcement of Siri back in 2011.
01:52And you'll see that it's actually, you know, I think it it aged pretty well.
02:00What is the weather like today?
02:06Here's the forecast for today.
02:09It is THAT EASY.
02:11[cheering]
02:12[applause]
02:13LOTS OF THINGS.
02:14We've integrated with the stocks.
02:15So, you can ask it about the stock market.
02:17Something like How is the NASDAQ doing today?
02:23NASDAQ composite is down right now at 2,321.70.
02:29Again, you can ask this from the lock screen anywhere.
02:31Just press the button and ask.
02:32You can ask about, you know, the NASDAQ, the Dow.
02:35So, what you just saw is what kind of a voice agent.
02:38It was a bit constrained, but it was technically a voice agent.
02:41And the architecture behind it, so you have seen a thousand times today the STT LLM TTS.
02:46Back then, it was even worse, right?
02:48So, there was no LLM, obviously.
02:50So, there was what was called natural language understanding.
02:53So, you would go from the transcript and uh try to do basic classification of what is expected, uh what is the app that is supposed to be uh controlled,
03:03what is the action to trigger, and so on.
03:04So, it was a very complex pipeline and very constrained to very specific use cases.
03:11So, it was what was called close uh ended um uh dialogue back then.
03:18Then, the next generation of models, uh I could say, I think was is illustrated pretty well by the OpenAI voice mode, not the advanced voice mode, the original one.
03:29In which case, you just have um a conversational experience that is open-ended.
03:33So, it's much less constrained than Siri.
03:35You can talk about pretty much anything.
03:37At the same time, you don't have um as the same agentic capabilities, let's say, in the sense that it cannot really give you the NASDAQ or uh give you the weather.
03:51We want to hear a bedtime story.
03:53Tell us a story about the super-duper sunflower hedgehog named Larry.
03:57Start with telling us a little bit about him.
04:03Larry was a unique hedgehog unlike any other.
04:06He had bright sunflower petals instead of spines.
04:09Residing in Meadowville, Larry spread joy and color wherever he rolled.
04:15So, in that context, basically that was, you know, kind of second generation of models where all the logic, the very complex logic was replaced uh by an LLM.
04:24And if we compare Siri and the voice mode from OpenAI, uh not the Ad Ons one, Ad Ons one, the original one, uh the big strength is opening the dialogue,
04:36but the agentic capabilities were inexistent.
04:38The naturalness was, obviously, much better than with Siri.
04:41The latency was still pretty bad.
04:43So, we are talking about several seconds, and that was documented in the in the previous presentation.
04:48And so, then, you know, we're in this third phase, which is a voice, what we call voice agents.
04:55Uh and here I'm going to show a quick demo of uh of a drive-thru bot uh live-coded with our with our voice models.
05:03Hi there.
05:03What can I get started for you today?
05:06Hi, I'm actually in a hurry and I'd like to order a chicken sandwich quickly.
05:11We have the classic, spicy, deluxe, or grilled chicken sandwich.
05:15Which one would you like?
05:17Uh let's go with the classic chicken sandwich.
05:19Anything else for you?
05:21Uh what else do you have on the menu?
05:24We have sandwiches, sides, drinks, and desserts.
05:28What can I get for you?
05:30Can I get some mac and cheese, maybe?
05:32You have a classic chicken sandwich and mac and cheese.
05:36Your total comes to eight.
05:38So, now you know, unlike what we just saw, it's an actual agent.
05:42It's taking actions.
05:43It's keeping track of the order.
05:45It's then going to make you pay.
05:46So, it's it's an actual voice agent that can do uh real tasks.
05:50So, here instead of having an LLM that is just a conversational interface, we have a real agent that is empowered with tool call, reasoning, planning, and and all this stuff.
06:00So, what we see now is we have gained back agentic capabilities, and actually they are much more uh powerful and generic than before, while keeping a very good level of uh of naturalness.
06:16And that's where speech-to-speech LLM came.
06:19In particular, what we could see here is the latency, it's better with cascaded system, but it's still higher than you will have with human conversation.
06:29And as also was explained before, the naturalness is fundamentally limited by the fact that you go through text, so you lose a lot of information about what uh is said, the tone, the emotion of the user, and so on and so forth.
06:41So, now that we have tackled intelligence and agentic capabilities, speech-to-speech seems like a natural next step for naturalness and latency.
06:48And so here it's the announcement from the uh OpenAI advanced voice mode.
06:53[clears throat]
06:53Hey, ChatGPT.
06:54I'm Mark.
06:55How are you?
06:56Oh, Mark.
06:58I'm doing great.
06:59Thanks for asking.
07:00How about you?
07:02Hey, so I'm on stage right now.
07:03I'm doing a live demo, and frankly I'm feeling a little bit nervous.
07:06Can you help me calm my nerves a little bit?
07:09Oh, you're doing a live demo right now?
07:12That's awesome.
07:13Just
07:14I think we all remember it was very impressive very impressive release.
07:18And in that context now, all the steps of STT, LLM, and TTS have been absorbed into a a single one.
07:27And so now, intelligence, you know, like naturalness is still very good.
07:33Uh actually it can be better because it can understand non-linguistic information.
07:37Latency is really, really nice.
07:40Honestly, it doesn't make sense to go uh better than that.
07:44Interestingly and everyone was used any uh speech-to-speech model can uh attest that the intelligence is still much more limited in that context than uh the cascaded counterpart.
07:57So, the speech-to-speech models are fundamentally still limited compared to the textual models.
08:03Another limitation is turn-taking.
08:05So, people tend to mix speech-to-speech and full duplex.
08:11But basically, when you do have a speech-to-speech model like GPT-3 time, it's still based on fundamental turn-taking.
08:17In the sense that it's going to segment the conversation into as long as the model is speaking or the model is listening.
08:23And to give to show you how this can make an interaction unnatural, I'm going to show a a small demo with what is called backchanneling, which is this very human thing that you do when someone talks to you is that you say, "Mhm, yeah."
08:35and so on.
08:37Hey, how's it going?
08:38Just like to brainstorm a bit about it with you.
08:42Oh, that's a great topic.
08:43Yeah, I'd love to help you brainstorm.
08:45Are you thinking what Exactly.
08:48Yeah, I was thinking
08:48No, no, I didn't mean to interrupt, you know, I was just saying, "Yeah."
08:51like that.
08:51You can just keep going.
08:52I know, don't mind me.
08:53It's just something I typically do.
08:56Uh no worries at all.
08:57Yeah, I was just going to say we could break it down into a few aspects.
09:00Like Yeah, exactly.
09:02please stop stop interrupting.
09:03You know, it's called backchanneling.
09:04Humans do it all the time.
09:05It shows that you're just following the conversation.
09:07That you don't, you know, like interrupt you in your flow.
09:10Just just going.
09:12Ah, got it.
09:13Thanks for letting me know.
09:14No problem.
09:16Oh, come on.
09:17Yeah, so you see, you know, it's it's still very annoying.
09:19Uh you can have lightning speed latency.
09:22Fundamentally, this is uh an issue that can not be resolved when you're using turn-taking.
09:28So, here that's the walkie-talkie.
09:30Um any real-time model today, I mean, now there is a bidirectional one that will come from OpenAI, but it's called half duplex.
09:38So, the model is listening or speaking.
09:40A human conversation has a constant flow between two people.
09:45People do back channeling.
09:46People interrupt one another, talk on one another, and so on.
09:50If you have If you're having a relative on the phone, there is up to 20% of the time where you are both speaking at the same time.
09:57And that makes, you know, this very flexible dynamics in the conversation makes it much more comfortable for humans.
10:04And so, to understand how we can make a model full duplex, I'll give a very short presentation of how we train such models.
10:14So, the way you create a speech-to-speech model half duplex or full duplex is the following one.
10:18So, you you start from a text LLM, which is a probabilistic models over over words.
10:23And instead of predicting the next word based on the past, what you want to do is rather predict the next audio based based on the past audio.
10:31The issue now is that if you pass a raw audio to your model, which is, you know, a waveform, it's air pressure variations.
10:39Uh basically, you take this sentence, it's eight words.
10:44It takes around 3 seconds to pronounce it.
10:46And so, at 24 kHz audio, instead of having eight words, the audio form is 72,000 time steps that you would need to feed to your LLM.
10:55Given that LLMs have quadratic complexity with sequence length, so the complexity is the square of the sequence length.
11:01A 10,000 times longer sequence is 100 million times more expensive to to process.
11:06So, there is no way you can train an LLM on raw audio.
11:08So, the way you address it is by creating neural codecs, or you can also call them audio tokenizers.
11:14And basically, it's an encoder that takes an audio and compresses it in a very dense compressed representation, a bit similar to text.
11:22And then you have a decoder that can reconstruct high-quality audio from it.
11:26So, now you have gone from the audio domain into a abstract representation domain, where you can train an LLM exactly like you would train it on text.
11:35And the speech-to-speech model from ElevenLabs, as I was showing before, works in this fashion.
11:39So, instead of having text tokens into your model, you have audio tokens that represent either the LLM or the user, and you put them one after the other, and the model predicts the audio tokens
11:51uh that should be said by the model, being given the context from both sides of the conversation.
11:56However, you can see that it's still a sequence between user and system, which is still half duplex.
12:03So, how did we make the first full duplex model ever?
12:06Very simple.
12:07We call it multi-stream language models.
12:09That's the technology now used also by Thinking Machines for their interaction model, and most likely by the for the by the directional model of OpenAI.
12:17Is that instead of having a transformer that models one sequence of tokens, it models two of them, so that both parties can be active at the same time, inactive at the same time, one active and one inactive.
12:29And I just show a very quick demo of uh of how it sounds
12:33[snorts]
12:33like, but that's the release of machine August 2024, uh where we did an announcement live on stage talking to it for the first time.
12:41And you'll see that the model often guesses the end of the question, answers over the speaker, and both speaking at the same time is not breaking the flow like we saw with
12:52GPT.
12:53The whole thing is just extremely resilient to the most chaotic uh situations.
12:57So, the planet is serious 22.
13:00Can you plot a trajectory course to it, please?
13:02Yes, sir.
13:03Okay.
13:04How long is it going to take us to get there?
13:06it out.
13:06It's approximately 5 months to get there.
13:09Okay, that's that's not too bad.
13:11Uh do you think we have all we need on board the ship to start the mission?
13:14We have everything we need.
13:16So, back then it was even a bit irritating to people because it was interrupting you all the time.
13:21But the thing is that you can use you could still use it in extremely noisy environments with a lot of noise, people coughing, and so on.
13:27And you know, the flow is just constant.
13:29You don't get this very irritating break of the conversational flow.
13:33So, these full duplex models, they are the highest level of naturalness you can expect.
13:39That's the same conversation with a human.
13:42The thing is, in with our models, it was even more stupid than speech-to-speech models that were already less intelligent than cascaded systems.
13:50It's probably fine for some use cases if you just want to have a chit-chat.
13:53You know, the model doesn't need to be very intelligent.
13:55But make an actual full duplex voice agent, there is no way we can give up on on intelligence just to gain uh speech-to-speech abilities.
14:04So, how do we finally make models that tackle all these aspects jointly?
14:08And I think interestingly, if you if you look at the history I showed, there is a tension between naturalness and intelligence.
14:14So, every time we improve naturalness or humanness of the of the models, they were less intelligent than the cascaded system.
14:22The cascaded agents, they are basically as smart as the best text models.
14:25So, if you have a voice agent that is powered by the latest model from Anthropic or OpenAI, it's going to be extremely smart, have all the same reliability for tool call, and so on.
14:34Speech-to-speech has this naturalness aspect.
14:37However, you give up intelligence to get that.
14:40And the reason why you give up intelligence is remember that the LLM is a model that has a certain number of weights that we call the capacity.
14:48And if you take a text model and now it not only has to handle text, but it also needs to understand speech and produce speech, it's taking some of its capacity,
14:58and this capacity now is taken from the intelligence.
15:01So, fundamentally, there is a cost of adding a new modality to a text model that is going to be paid in in intelligence.
15:08So, where do we go from here?
15:10There are two paths that are in front of us, and both are going to be explored at the same time.
15:14The first one is scaling the model.
15:16So, making your speech-to-speech model bigger, better pre-trained, better post-trained, and so on.
15:23We likely progressively increase its intelligence until it it's good enough for a lot of use cases.
15:29The second one is splitting the model between naturalness and intelligence.
15:35The first one, I'm not at OpenAI, so I don't know because they don't release their model.
15:39I guess OpenAI is the path one, so it's a frontier text LLM with a lot of science around post-training, in- instruct tuning to fine-tune it on audio, and teaching it to be quite smart while using audio.
15:54The nice thing about that is you have a single model to orchestrate, so it's quite easy to deploy.
15:59Um and one big aspect, however, is that it's a extremely complex and costly process to go from the text model to the speech-to-speech model.
16:11The second path is to split it.
16:12It's an approach that we introduced in one of our recent papers called Moushiraq, and that has been reused by, in particular, the Thinking Machine Interaction models.
16:21Where, basically, the idea is that now you have two models.
16:24The first one is a small, maybe even on-device, full-duplex, extremely natural speech-to-speech interface.
16:30And its only role is to keep a very natural conversation and be able to delegate all the thinking, tool calling, reasoning, agentic capabilities to a background text model.
16:43And so, the way to see it is you have a background text LLM that receives asynchronously queries from hundreds to thousands of small voice interfaces and just give them their text, you know?
16:54And, basically, what we did is um very small full-duplex model that just needs to know when it doesn't know, so that it can delegate to the background model.
17:04And the reason why we believe mostly in this approach, um and I go back to to it later, it's a our uh let's say our culture is more of first one, the bitter lesson.
17:17So, every time we've been pushing for end-to-end systems and so on.
17:21But now I think the hybrid approach has two main advantages.
17:25The The first one is cost.
17:26So, speech-to-speech models are notoriously quite expensive.
17:30And when you think about it, it's a loss of money to do chit-chat with gigantic speech-to-speech models that can resolve differential equations and so on.
17:40So, it doesn't really make sense economically to get all your workflow through this gigantic multimodal mixture of experts.
17:48At the same time, we see that people are very attached to their ability to control the backend, to be able to switch So they So they 5 was released a few minutes ago.
17:57People want to switch the backend and the intelligence and get a lot of optionality on that, right?
18:02When you're using a speech-to-speech model, your your hands are a bit tied with this model provider.
18:07And to give you an idea of that, until recently the AdSense voice mode from OpenAI was powered by GPT-4o, despite the fact that there have been several generations of the text model since then, because this process is so expensive
18:19and so long.
18:21For this reason, we rather bet on the hybrid approach because that will give something that is not only very natural and very nice for demos and impressive, but also will be a viable
18:33alternative from a economic point of view and agentic capabilities point of view to the best cascaded systems that are still most of the market today in voice.
18:43So, what now?
18:45Uh you can use our models on gradium.ai.
18:48You can apply to gradium.
18:49We are recruiting research scientists and engineers.
18:53And thanks for your attention.
18:56[applause]
19:10[music]