WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=a8EcVumh71E

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/a8EcVumh71E.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:14.520
Okay, hi everyone.

s3
00:00:14.520 --> 00:00:18.360
I'm Nel co-founder and CEO of of Gradio.

s4
00:00:18.360 --> 00:00:22.000
So Gradio is a startup based in Paris.

s5
00:00:22.000 --> 00:00:24.440
Most of our background is from research.

s6
00:00:24.440 --> 00:00:32.720
In particular, we have invented algorithms such as audio LLMs, speech speech-to-speech models, neural codex, and so on and so forth.

s7
00:00:32.720 --> 00:00:41.800
And basically we started from a research project called QTI, a non-profit research lab that has been focusing on voice since day one.

s8
00:00:41.800 --> 00:00:56.200
So in particular, we released in 2024 the first full duplex speech-to-speech model called Moshi, the first real-time speech-to-speech translation system called Hibiki, and the first TTS model that can run locally on a on a smartphone.

s9
00:00:56.200 --> 00:01:04.760
And basically I will just say a few words about what we do, but we are a model company that trains models for building voice agents and voice applications.

s10
00:01:04.760 --> 00:01:11.280
So we do TTS, API, and on-device speech-to-text, speech-to-speech translation, and much more to come.

s11
00:01:11.280 --> 00:01:17.080
What we do is that we train foundation models for audio, and then we can apply them for a lot of different tasks.

s12
00:01:17.080 --> 00:01:32.960
So I go quickly on voice agents because it's the fourth talk about the topic, but basically now we have these voice interfaces that we can use to do a lot of things across a variety of products and types of interactions with NPCs,

s13
00:01:32.960 --> 00:01:37.840
with customer agents, language learners, coach, and so on and so forth.

s14
00:01:37.840 --> 00:01:46.360
And in this talk I tried to go through the history of this technology and where I see it going in the next years.

s15
00:01:46.360 --> 00:01:52.920
And maybe to start, I think we can take a look at the announcement of Siri back in 2011.

s16
00:01:52.920 --> 00:01:59.640
And you'll see that it's actually, you know, I think it it aged pretty well.

s17
00:02:00.280 --> 00:02:03.120
What is the weather like today?

s18
00:02:06.400 --> 00:02:09.479
Here's the forecast for today.

s19
00:02:09.479 --> 00:02:11.001
It is THAT EASY.

s20
00:02:11.001 --> 00:02:12.311
[cheering]

s21
00:02:12.311 --> 00:02:13.440
[applause]

s22
00:02:13.440 --> 00:02:14.120
LOTS OF THINGS.

s23
00:02:14.120 --> 00:02:15.480
We've integrated with the stocks.

s24
00:02:15.480 --> 00:02:17.000
So, you can ask it about the stock market.

s25
00:02:17.000 --> 00:02:22.040
Something like How is the NASDAQ doing today?

s26
00:02:23.720 --> 00:02:28.240
NASDAQ composite is down right now at 2,321.70.

s27
00:02:29.480 --> 00:02:31.440
Again, you can ask this from the lock screen anywhere.

s28
00:02:31.440 --> 00:02:32.360
Just press the button and ask.

s29
00:02:32.360 --> 00:02:35.240
You can ask about, you know, the NASDAQ, the Dow.

s30
00:02:35.240 --> 00:02:38.800
So, what you just saw is what kind of a voice agent.

s31
00:02:38.800 --> 00:02:41.440
It was a bit constrained, but it was technically a voice agent.

s32
00:02:41.440 --> 00:02:46.440
And the architecture behind it, so you have seen a thousand times today the STT LLM TTS.

s33
00:02:46.440 --> 00:02:48.120
Back then, it was even worse, right?

s34
00:02:48.120 --> 00:02:50.440
So, there was no LLM, obviously.

s35
00:02:50.440 --> 00:02:53.120
So, there was what was called natural language understanding.

s36
00:02:53.120 --> 00:03:03.000
So, you would go from the transcript and uh try to do basic classification of what is expected, uh what is the app that is supposed to be uh controlled,

s37
00:03:03.000 --> 00:03:04.640
what is the action to trigger, and so on.

s38
00:03:04.640 --> 00:03:11.200
So, it was a very complex pipeline and very constrained to very specific use cases.

s39
00:03:11.200 --> 00:03:18.280
So, it was what was called close uh ended um uh dialogue back then.

s40
00:03:18.280 --> 00:03:29.000
Then, the next generation of models, uh I could say, I think was is illustrated pretty well by the OpenAI voice mode, not the advanced voice mode, the original one.

s41
00:03:29.000 --> 00:03:33.600
In which case, you just have um a conversational experience that is open-ended.

s42
00:03:33.600 --> 00:03:35.400
So, it's much less constrained than Siri.

s43
00:03:35.400 --> 00:03:37.320
You can talk about pretty much anything.

s44
00:03:37.320 --> 00:03:48.080
At the same time, you don't have um as the same agentic capabilities, let's say, in the sense that it cannot really give you the NASDAQ or uh give you the weather.

s45
00:03:51.000 --> 00:03:53.160
We want to hear a bedtime story.

s46
00:03:53.160 --> 00:03:57.400
Tell us a story about the super-duper sunflower hedgehog named Larry.

s47
00:03:57.400 --> 00:04:00.720
Start with telling us a little bit about him.

s48
00:04:03.440 --> 00:04:06.640
Larry was a unique hedgehog unlike any other.

s49
00:04:06.640 --> 00:04:09.880
He had bright sunflower petals instead of spines.

s50
00:04:09.880 --> 00:04:15.040
Residing in Meadowville, Larry spread joy and color wherever he rolled.

s51
00:04:15.040 --> 00:04:24.320
So, in that context, basically that was, you know, kind of second generation of models where all the logic, the very complex logic was replaced uh by an LLM.

s52
00:04:24.320 --> 00:04:36.200
And if we compare Siri and the voice mode from OpenAI, uh not the Ad Ons one, Ad Ons one, the original one, uh the big strength is opening the dialogue,

s53
00:04:36.200 --> 00:04:38.880
but the agentic capabilities were inexistent.

s54
00:04:38.880 --> 00:04:41.919
The naturalness was, obviously, much better than with Siri.

s55
00:04:41.919 --> 00:04:43.240
The latency was still pretty bad.

s56
00:04:43.240 --> 00:04:48.880
So, we are talking about several seconds, and that was documented in the in the previous presentation.

s57
00:04:48.880 --> 00:04:55.640
And so, then, you know, we're in this third phase, which is a voice, what we call voice agents.

s58
00:04:55.640 --> 00:05:03.040
Uh and here I'm going to show a quick demo of uh of a drive-thru bot uh live-coded with our with our voice models.

s59
00:05:03.040 --> 00:05:03.960
Hi there.

s60
00:05:03.960 --> 00:05:06.680
What can I get started for you today?

s61
00:05:06.680 --> 00:05:11.360
Hi, I'm actually in a hurry and I'd like to order a chicken sandwich quickly.

s62
00:05:11.360 --> 00:05:15.880
We have the classic, spicy, deluxe, or grilled chicken sandwich.

s63
00:05:15.880 --> 00:05:17.200
Which one would you like?

s64
00:05:17.200 --> 00:05:19.720
Uh let's go with the classic chicken sandwich.

s65
00:05:19.720 --> 00:05:21.480
Anything else for you?

s66
00:05:21.480 --> 00:05:24.320
Uh what else do you have on the menu?

s67
00:05:24.320 --> 00:05:28.320
We have sandwiches, sides, drinks, and desserts.

s68
00:05:28.320 --> 00:05:30.080
What can I get for you?

s69
00:05:30.080 --> 00:05:32.840
Can I get some mac and cheese, maybe?

s70
00:05:32.840 --> 00:05:36.120
You have a classic chicken sandwich and mac and cheese.

s71
00:05:36.120 --> 00:05:38.000
Your total comes to eight.

s72
00:05:38.000 --> 00:05:42.200
So, now you know, unlike what we just saw, it's an actual agent.

s73
00:05:42.200 --> 00:05:43.200
It's taking actions.

s74
00:05:43.200 --> 00:05:45.000
It's keeping track of the order.

s75
00:05:45.000 --> 00:05:46.520
It's then going to make you pay.

s76
00:05:46.520 --> 00:05:50.800
So, it's it's an actual voice agent that can do uh real tasks.

s77
00:05:50.800 --> 00:06:00.160
So, here instead of having an LLM that is just a conversational interface, we have a real agent that is empowered with tool call, reasoning, planning, and and all this stuff.

s78
00:06:00.160 --> 00:06:16.360
So, what we see now is we have gained back agentic capabilities, and actually they are much more uh powerful and generic than before, while keeping a very good level of uh of naturalness.

s79
00:06:16.360 --> 00:06:19.560
And that's where speech-to-speech LLM came.

s80
00:06:19.560 --> 00:06:29.200
In particular, what we could see here is the latency, it's better with cascaded system, but it's still higher than you will have with human conversation.

s81
00:06:29.200 --> 00:06:41.000
And as also was explained before, the naturalness is fundamentally limited by the fact that you go through text, so you lose a lot of information about what uh is said, the tone, the emotion of the user, and so on and so forth.

s82
00:06:41.000 --> 00:06:48.120
So, now that we have tackled intelligence and agentic capabilities, speech-to-speech seems like a natural next step for naturalness and latency.

s83
00:06:48.120 --> 00:06:53.309
And so here it's the announcement from the uh OpenAI advanced voice mode.

s84
00:06:53.309 --> 00:06:53.720
[clears throat]

s85
00:06:53.720 --> 00:06:54.760
Hey, ChatGPT.

s86
00:06:54.760 --> 00:06:55.480
I'm Mark.

s87
00:06:55.480 --> 00:06:56.800
How are you?

s88
00:06:56.800 --> 00:06:58.640
Oh, Mark.

s89
00:06:58.640 --> 00:06:59.680
I'm doing great.

s90
00:06:59.680 --> 00:07:00.680
Thanks for asking.

s91
00:07:00.680 --> 00:07:02.000
How about you?

s92
00:07:02.000 --> 00:07:03.800
Hey, so I'm on stage right now.

s93
00:07:03.800 --> 00:07:06.919
I'm doing a live demo, and frankly I'm feeling a little bit nervous.

s94
00:07:06.919 --> 00:07:09.840
Can you help me calm my nerves a little bit?

s95
00:07:09.840 --> 00:07:12.200
Oh, you're doing a live demo right now?

s96
00:07:12.200 --> 00:07:13.880
That's awesome.

s97
00:07:13.880 --> 00:07:14.600
Just

s98
00:07:14.600 --> 00:07:18.680
I think we all remember it was very impressive very impressive release.

s99
00:07:18.680 --> 00:07:27.120
And in that context now, all the steps of STT, LLM, and TTS have been absorbed into a a single one.

s100
00:07:27.120 --> 00:07:33.720
And so now, intelligence, you know, like naturalness is still very good.

s101
00:07:33.720 --> 00:07:37.919
Uh actually it can be better because it can understand non-linguistic information.

s102
00:07:37.919 --> 00:07:40.200
Latency is really, really nice.

s103
00:07:40.200 --> 00:07:44.080
Honestly, it doesn't make sense to go uh better than that.

s104
00:07:44.080 --> 00:07:57.400
Interestingly and everyone was used any uh speech-to-speech model can uh attest that the intelligence is still much more limited in that context than uh the cascaded counterpart.

s105
00:07:57.400 --> 00:08:03.200
So, the speech-to-speech models are fundamentally still limited compared to the textual models.

s106
00:08:03.200 --> 00:08:05.680
Another limitation is turn-taking.

s107
00:08:05.680 --> 00:08:11.480
So, people tend to mix speech-to-speech and full duplex.

s108
00:08:11.480 --> 00:08:17.760
But basically, when you do have a speech-to-speech model like GPT-3 time, it's still based on fundamental turn-taking.

s109
00:08:17.760 --> 00:08:23.400
In the sense that it's going to segment the conversation into as long as the model is speaking or the model is listening.

s110
00:08:23.400 --> 00:08:35.320
And to give to show you how this can make an interaction unnatural, I'm going to show a a small demo with what is called backchanneling, which is this very human thing that you do when someone talks to you is that you say, "Mhm, yeah."

s111
00:08:35.320 --> 00:08:37.560
and so on.

s112
00:08:37.560 --> 00:08:38.719
Hey, how's it going?

s113
00:08:38.719 --> 00:08:42.039
Just like to brainstorm a bit about it with you.

s114
00:08:42.039 --> 00:08:43.400
Oh, that's a great topic.

s115
00:08:43.400 --> 00:08:45.320
Yeah, I'd love to help you brainstorm.

s116
00:08:45.320 --> 00:08:48.040
Are you thinking what Exactly.

s117
00:08:48.040 --> 00:08:48.720
Yeah, I was thinking

s118
00:08:48.720 --> 00:08:51.400
No, no, I didn't mean to interrupt, you know, I was just saying, "Yeah."

s119
00:08:51.400 --> 00:08:51.800
like that.

s120
00:08:51.800 --> 00:08:52.920
You can just keep going.

s121
00:08:52.920 --> 00:08:53.880
I know, don't mind me.

s122
00:08:53.880 --> 00:08:56.240
It's just something I typically do.

s123
00:08:56.240 --> 00:08:57.560
Uh no worries at all.

s124
00:08:57.560 --> 00:09:00.400
Yeah, I was just going to say we could break it down into a few aspects.

s125
00:09:00.400 --> 00:09:02.240
Like Yeah, exactly.

s126
00:09:02.240 --> 00:09:03.560
please stop stop interrupting.

s127
00:09:03.560 --> 00:09:04.760
You know, it's called backchanneling.

s128
00:09:04.760 --> 00:09:05.920
Humans do it all the time.

s129
00:09:05.920 --> 00:09:07.920
It shows that you're just following the conversation.

s130
00:09:07.920 --> 00:09:10.400
That you don't, you know, like interrupt you in your flow.

s131
00:09:10.400 --> 00:09:12.400
Just just going.

s132
00:09:12.400 --> 00:09:13.280
Ah, got it.

s133
00:09:13.280 --> 00:09:14.480
Thanks for letting me know.

s134
00:09:14.480 --> 00:09:16.160
No problem.

s135
00:09:16.160 --> 00:09:17.160
Oh, come on.

s136
00:09:17.160 --> 00:09:19.920
Yeah, so you see, you know, it's it's still very annoying.

s137
00:09:19.920 --> 00:09:22.480
Uh you can have lightning speed latency.

s138
00:09:22.480 --> 00:09:28.800
Fundamentally, this is uh an issue that can not be resolved when you're using turn-taking.

s139
00:09:28.800 --> 00:09:30.880
So, here that's the walkie-talkie.

s140
00:09:30.880 --> 00:09:38.280
Um any real-time model today, I mean, now there is a bidirectional one that will come from OpenAI, but it's called half duplex.

s141
00:09:38.280 --> 00:09:40.840
So, the model is listening or speaking.

s142
00:09:40.840 --> 00:09:45.480
A human conversation has a constant flow between two people.

s143
00:09:45.480 --> 00:09:46.680
People do back channeling.

s144
00:09:46.680 --> 00:09:50.240
People interrupt one another, talk on one another, and so on.

s145
00:09:50.240 --> 00:09:57.400
If you have If you're having a relative on the phone, there is up to 20% of the time where you are both speaking at the same time.

s146
00:09:57.400 --> 00:10:04.440
And that makes, you know, this very flexible dynamics in the conversation makes it much more comfortable for humans.

s147
00:10:04.440 --> 00:10:14.400
And so, to understand how we can make a model full duplex, I'll give a very short presentation of how we train such models.

s148
00:10:14.400 --> 00:10:18.400
So, the way you create a speech-to-speech model half duplex or full duplex is the following one.

s149
00:10:18.400 --> 00:10:23.320
So, you you start from a text LLM, which is a probabilistic models over over words.

s150
00:10:23.320 --> 00:10:31.360
And instead of predicting the next word based on the past, what you want to do is rather predict the next audio based based on the past audio.

s151
00:10:31.360 --> 00:10:39.520
The issue now is that if you pass a raw audio to your model, which is, you know, a waveform, it's air pressure variations.

s152
00:10:39.520 --> 00:10:44.480
Uh basically, you take this sentence, it's eight words.

s153
00:10:44.480 --> 00:10:46.520
It takes around 3 seconds to pronounce it.

s154
00:10:46.520 --> 00:10:55.240
And so, at 24 kHz audio, instead of having eight words, the audio form is 72,000 time steps that you would need to feed to your LLM.

s155
00:10:55.240 --> 00:11:01.640
Given that LLMs have quadratic complexity with sequence length, so the complexity is the square of the sequence length.

s156
00:11:01.640 --> 00:11:06.000
A 10,000 times longer sequence is 100 million times more expensive to to process.

s157
00:11:06.000 --> 00:11:08.680
So, there is no way you can train an LLM on raw audio.

s158
00:11:08.680 --> 00:11:14.680
So, the way you address it is by creating neural codecs, or you can also call them audio tokenizers.

s159
00:11:14.680 --> 00:11:22.520
And basically, it's an encoder that takes an audio and compresses it in a very dense compressed representation, a bit similar to text.

s160
00:11:22.520 --> 00:11:26.360
And then you have a decoder that can reconstruct high-quality audio from it.

s161
00:11:26.360 --> 00:11:35.560
So, now you have gone from the audio domain into a abstract representation domain, where you can train an LLM exactly like you would train it on text.

s162
00:11:35.560 --> 00:11:39.960
And the speech-to-speech model from ElevenLabs, as I was showing before, works in this fashion.

s163
00:11:39.960 --> 00:11:51.560
So, instead of having text tokens into your model, you have audio tokens that represent either the LLM or the user, and you put them one after the other, and the model predicts the audio tokens

s164
00:11:51.560 --> 00:11:56.920
uh that should be said by the model, being given the context from both sides of the conversation.

s165
00:11:56.920 --> 00:12:03.000
However, you can see that it's still a sequence between user and system, which is still half duplex.

s166
00:12:03.000 --> 00:12:06.240
So, how did we make the first full duplex model ever?

s167
00:12:06.240 --> 00:12:07.240
Very simple.

s168
00:12:07.240 --> 00:12:09.440
We call it multi-stream language models.

s169
00:12:09.440 --> 00:12:17.960
That's the technology now used also by Thinking Machines for their interaction model, and most likely by the for the by the directional model of OpenAI.

s170
00:12:17.960 --> 00:12:29.200
Is that instead of having a transformer that models one sequence of tokens, it models two of them, so that both parties can be active at the same time, inactive at the same time, one active and one inactive.

s171
00:12:29.200 --> 00:12:33.258
And I just show a very quick demo of uh of how it sounds

s172
00:12:33.258 --> 00:12:33.440
[snorts]

s173
00:12:33.440 --> 00:12:41.960
like, but that's the release of machine August 2024, uh where we did an announcement live on stage talking to it for the first time.

s174
00:12:41.960 --> 00:12:52.640
And you'll see that the model often guesses the end of the question, answers over the speaker, and both speaking at the same time is not breaking the flow like we saw with

s175
00:12:52.640 --> 00:12:53.280
GPT.

s176
00:12:53.280 --> 00:12:57.560
The whole thing is just extremely resilient to the most chaotic uh situations.

s177
00:12:57.560 --> 00:13:00.080
So, the planet is serious 22.

s178
00:13:00.080 --> 00:13:02.560
Can you plot a trajectory course to it, please?

s179
00:13:02.560 --> 00:13:03.760
Yes, sir.

s180
00:13:03.760 --> 00:13:04.360
Okay.

s181
00:13:04.360 --> 00:13:06.080
How long is it going to take us to get there?

s182
00:13:06.080 --> 00:13:06.839
it out.

s183
00:13:06.839 --> 00:13:09.160
It's approximately 5 months to get there.

s184
00:13:09.160 --> 00:13:11.320
Okay, that's that's not too bad.

s185
00:13:11.320 --> 00:13:14.640
Uh do you think we have all we need on board the ship to start the mission?

s186
00:13:14.640 --> 00:13:16.560
We have everything we need.

s187
00:13:16.560 --> 00:13:21.480
So, back then it was even a bit irritating to people because it was interrupting you all the time.

s188
00:13:21.480 --> 00:13:27.880
But the thing is that you can use you could still use it in extremely noisy environments with a lot of noise, people coughing, and so on.

s189
00:13:27.880 --> 00:13:29.560
And you know, the flow is just constant.

s190
00:13:29.560 --> 00:13:33.680
You don't get this very irritating break of the conversational flow.

s191
00:13:33.680 --> 00:13:39.400
So, these full duplex models, they are the highest level of naturalness you can expect.

s192
00:13:39.400 --> 00:13:42.080
That's the same conversation with a human.

s193
00:13:42.080 --> 00:13:50.520
The thing is, in with our models, it was even more stupid than speech-to-speech models that were already less intelligent than cascaded systems.

s194
00:13:50.520 --> 00:13:53.640
It's probably fine for some use cases if you just want to have a chit-chat.

s195
00:13:53.640 --> 00:13:55.920
You know, the model doesn't need to be very intelligent.

s196
00:13:55.920 --> 00:14:04.120
But make an actual full duplex voice agent, there is no way we can give up on on intelligence just to gain uh speech-to-speech abilities.

s197
00:14:04.120 --> 00:14:08.920
So, how do we finally make models that tackle all these aspects jointly?

s198
00:14:08.920 --> 00:14:14.800
And I think interestingly, if you if you look at the history I showed, there is a tension between naturalness and intelligence.

s199
00:14:14.800 --> 00:14:22.320
So, every time we improve naturalness or humanness of the of the models, they were less intelligent than the cascaded system.

s200
00:14:22.320 --> 00:14:25.920
The cascaded agents, they are basically as smart as the best text models.

s201
00:14:25.920 --> 00:14:34.000
So, if you have a voice agent that is powered by the latest model from Anthropic or OpenAI, it's going to be extremely smart, have all the same reliability for tool call, and so on.

s202
00:14:34.000 --> 00:14:37.120
Speech-to-speech has this naturalness aspect.

s203
00:14:37.120 --> 00:14:40.200
However, you give up intelligence to get that.

s204
00:14:40.200 --> 00:14:48.839
And the reason why you give up intelligence is remember that the LLM is a model that has a certain number of weights that we call the capacity.

s205
00:14:48.839 --> 00:14:58.080
And if you take a text model and now it not only has to handle text, but it also needs to understand speech and produce speech, it's taking some of its capacity,

s206
00:14:58.080 --> 00:15:01.520
and this capacity now is taken from the intelligence.

s207
00:15:01.520 --> 00:15:08.960
So, fundamentally, there is a cost of adding a new modality to a text model that is going to be paid in in intelligence.

s208
00:15:08.960 --> 00:15:10.440
So, where do we go from here?

s209
00:15:10.440 --> 00:15:14.920
There are two paths that are in front of us, and both are going to be explored at the same time.

s210
00:15:14.920 --> 00:15:16.960
The first one is scaling the model.

s211
00:15:16.960 --> 00:15:23.880
So, making your speech-to-speech model bigger, better pre-trained, better post-trained, and so on.

s212
00:15:23.880 --> 00:15:29.680
We likely progressively increase its intelligence until it it's good enough for a lot of use cases.

s213
00:15:29.680 --> 00:15:35.280
The second one is splitting the model between naturalness and intelligence.

s214
00:15:35.280 --> 00:15:39.520
The first one, I'm not at OpenAI, so I don't know because they don't release their model.

s215
00:15:39.520 --> 00:15:54.120
I guess OpenAI is the path one, so it's a frontier text LLM with a lot of science around post-training, in- instruct tuning to fine-tune it on audio, and teaching it to be quite smart while using audio.

s216
00:15:54.120 --> 00:15:59.120
The nice thing about that is you have a single model to orchestrate, so it's quite easy to deploy.

s217
00:15:59.120 --> 00:16:11.560
Um and one big aspect, however, is that it's a extremely complex and costly process to go from the text model to the speech-to-speech model.

s218
00:16:11.560 --> 00:16:12.920
The second path is to split it.

s219
00:16:12.920 --> 00:16:21.000
It's an approach that we introduced in one of our recent papers called Moushiraq, and that has been reused by, in particular, the Thinking Machine Interaction models.

s220
00:16:21.000 --> 00:16:24.320
Where, basically, the idea is that now you have two models.

s221
00:16:24.320 --> 00:16:30.839
The first one is a small, maybe even on-device, full-duplex, extremely natural speech-to-speech interface.

s222
00:16:30.839 --> 00:16:43.160
And its only role is to keep a very natural conversation and be able to delegate all the thinking, tool calling, reasoning, agentic capabilities to a background text model.

s223
00:16:43.160 --> 00:16:54.560
And so, the way to see it is you have a background text LLM that receives asynchronously queries from hundreds to thousands of small voice interfaces and just give them their text, you know?

s224
00:16:54.560 --> 00:17:04.400
And, basically, what we did is um very small full-duplex model that just needs to know when it doesn't know, so that it can delegate to the background model.

s225
00:17:04.400 --> 00:17:17.319
And the reason why we believe mostly in this approach, um and I go back to to it later, it's a our uh let's say our culture is more of first one, the bitter lesson.

s226
00:17:17.319 --> 00:17:21.240
So, every time we've been pushing for end-to-end systems and so on.

s227
00:17:21.240 --> 00:17:25.520
But now I think the hybrid approach has two main advantages.

s228
00:17:25.520 --> 00:17:26.880
The The first one is cost.

s229
00:17:26.880 --> 00:17:30.440
So, speech-to-speech models are notoriously quite expensive.

s230
00:17:30.440 --> 00:17:40.280
And when you think about it, it's a loss of money to do chit-chat with gigantic speech-to-speech models that can resolve differential equations and so on.

s231
00:17:40.280 --> 00:17:48.680
So, it doesn't really make sense economically to get all your workflow through this gigantic multimodal mixture of experts.

s232
00:17:48.680 --> 00:17:57.320
At the same time, we see that people are very attached to their ability to control the backend, to be able to switch So they So they 5 was released a few minutes ago.

s233
00:17:57.320 --> 00:18:02.240
People want to switch the backend and the intelligence and get a lot of optionality on that, right?

s234
00:18:02.240 --> 00:18:07.600
When you're using a speech-to-speech model, your your hands are a bit tied with this model provider.

s235
00:18:07.600 --> 00:18:19.920
And to give you an idea of that, until recently the AdSense voice mode from OpenAI was powered by GPT-4o, despite the fact that there have been several generations of the text model since then, because this process is so expensive

s236
00:18:19.920 --> 00:18:21.920
and so long.

s237
00:18:21.920 --> 00:18:33.360
For this reason, we rather bet on the hybrid approach because that will give something that is not only very natural and very nice for demos and impressive, but also will be a viable

s238
00:18:33.360 --> 00:18:43.080
alternative from a economic point of view and agentic capabilities point of view to the best cascaded systems that are still most of the market today in voice.

s239
00:18:43.080 --> 00:18:45.840
So, what now?

s240
00:18:45.840 --> 00:18:48.040
Uh you can use our models on gradium.ai.

s241
00:18:48.040 --> 00:18:49.680
You can apply to gradium.

s242
00:18:49.680 --> 00:18:53.000
We are recruiting research scientists and engineers.

s243
00:18:53.000 --> 00:18:55.880
And thanks for your attention.

s244
00:18:56.438 --> 00:18:58.438
[applause]

s245
00:19:10.848 --> 00:19:12.848
[music]