WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=18Um2VjMM_g

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/18Um2VjMM_g.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:13.440
Hi everyone.

s3
00:00:13.440 --> 00:00:14.760
Thanks for coming to our talk.

s4
00:00:14.760 --> 00:00:22.320
We're going to talk a lot about how we are planning on voiceifying the agentic future with speech-to-speech research here at Google DeepMind.

s5
00:00:22.320 --> 00:00:23.200
My name is Valeria.

s6
00:00:23.200 --> 00:00:27.000
I'm the product lead for the speech-to-speech model in Gemini and Tom.

s7
00:00:27.000 --> 00:00:28.160
Yeah, my name is Tom.

s8
00:00:28.160 --> 00:00:31.240
I work on speech-to-speech as an engineer and Gemini.

s9
00:00:31.240 --> 00:00:33.960
So, yeah, great to be presenting our work.

s10
00:00:33.960 --> 00:00:35.040
Cool.

s11
00:00:35.040 --> 00:00:47.240
So, before we start diving into the history of Google research and audio and the latest research we've been doing, I just want to take a step back and really kind of like think about why this is important to us in the team.

s12
00:00:47.240 --> 00:00:53.800
And the main reason is that voice is the most natural way for humans to interact with both the physical and the virtual world.

s13
00:00:53.800 --> 00:00:58.200
And we're already seeing so many applications that are starting even within the own Google products.

s14
00:00:58.200 --> 00:01:05.680
We see voice being used to ask questions and you know, do homework or info seeking or EDU in search live and Gemini live.

s15
00:01:05.680 --> 00:01:11.680
All the way to these same models being deployed in cloud and the API for enterprise or voice agent use cases.

s16
00:01:11.680 --> 00:01:15.240
Should I use this as a microphone?

s17
00:01:15.640 --> 00:01:16.160
Hello.

s18
00:01:16.160 --> 00:01:16.880
Okay.

s19
00:01:16.880 --> 00:01:30.240
Um So, because of that, we only think that the number of applications is going to, you know, exponentially increase over the next few years and we believe that speech-to-speech models are the way to go when we want to build robust universal voice agents.

s20
00:01:30.240 --> 00:01:33.240
And with that, I'll hand it off to Tom.

s21
00:01:33.560 --> 00:01:33.840
All right.

s22
00:01:33.840 --> 00:01:41.640
So, of course, like one way to create a voice agent is to just add an ASR like speech-to-text model in the pipeline and then a text-to-speech model on the other end, right?

s23
00:01:41.640 --> 00:01:44.280
So, of course, that's where a lot of the history of speech come from.

s24
00:01:44.280 --> 00:01:46.840
Of course, Google has been working on this for a long time.

s25
00:01:46.840 --> 00:01:54.440
I'm going to give you a little bit of a historical overview of what speech modeling, especially speech-to-text automatic speech recognition, typically looks like, right?

s26
00:01:54.440 --> 00:01:58.360
So, up until around 2018, this usually involved a lot of different components.

s27
00:01:58.360 --> 00:02:08.119
You have um you know of course feature extraction that's fairly general and then you have all these different pieces like acoustic modeling, pronunciation modeling, language modeling, a second pass rescoring

s28
00:02:08.119 --> 00:02:12.000
that allows you to go from the audio input to a text transcription.

s29
00:02:12.000 --> 00:02:12.160
Right?

s30
00:02:12.160 --> 00:02:15.600
And of course like around 2018, these moved more and more towards end-to-end systems.

s31
00:02:15.600 --> 00:02:17.240
You don't have to have so much domain knowledge.

s32
00:02:17.240 --> 00:02:24.560
You can actually have mostly the the neural model learn this pattern and mapping between acoustic inputs and text.

s33
00:02:24.560 --> 00:02:27.280
But these aren't really the end-to-end models that we think about when we think about LLMs.

s34
00:02:27.280 --> 00:02:31.400
They're only doing kind of one thing, which is speech to a transcription of that speech.

s35
00:02:31.400 --> 00:02:33.400
They're not responding, they're not translating.

s36
00:02:33.400 --> 00:02:38.080
If you wanted to have the model tell you about the tone or the emotion or the speed of the speaker.

s37
00:02:38.080 --> 00:02:47.560
If you wanted to bias it towards words or much less like images, those are all things you have to build yourself as part of the system and there's really really a barrier to how easily you can scale

s38
00:02:47.560 --> 00:02:48.080
these systems.

s39
00:02:48.080 --> 00:02:52.480
So, fast forward to now, which is kind of of course the LLM era, right?

s40
00:02:52.480 --> 00:03:02.440
Like you know, the first LLMs were mostly text, but even there I think you could kind of hack audio embeddings into these text models and it kind of worked, but now of course for a long time now, Gemini models have been very natively

s41
00:03:02.440 --> 00:03:03.120
multimodal.

s42
00:03:03.120 --> 00:03:04.200
So, what does that mean?

s43
00:03:04.200 --> 00:03:10.800
It means that when we train these models in pre-training, which is where the bulk of the data comes from, these are multimodal interleaved examples, right?

s44
00:03:10.800 --> 00:03:15.080
So, the bottom uh diagram here gives you kind of one example of what that might look like.

s45
00:03:15.080 --> 00:03:24.040
So, this is a task where you're asking the model to summarize a bedtime story and there's a text prompt in the beginning, but then there's this sequence of video and audio

s46
00:03:24.040 --> 00:03:33.800
inputs that the model gets and then of course like what you expect the model to do here is produce um both the summary and also annotate timestamps for where interesting things happen and so forth.

s47
00:03:33.800 --> 00:03:39.959
So, this example is teaching the model to translate its understanding of the audio and the video into text.

s48
00:03:39.959 --> 00:03:43.480
You might have other examples in pre-training that ask you to caption a video.

s49
00:03:43.480 --> 00:03:51.000
So, you might have a video that has audio and the model is learning to bias towards both the video and the audio signal to caption this well.

s50
00:03:51.000 --> 00:03:55.280
And of course like there's limitless like YouTube videos with captions that you can train these models on.

s51
00:03:55.280 --> 00:03:58.920
Other models might actually try to generate audio from the video from the text, right?

s52
00:03:58.920 --> 00:04:08.440
So, you can have ASR, TTS, or any combination of these plus all of these sort of agentic tasks all kind of learned under one unified token embedding space.

s53
00:04:08.440 --> 00:04:19.600
So, this becomes a foundation for a lot of what we want to do in audio because we already have a model that understands audio, video, text, and how these things relate and transition from one to the next.

s54
00:04:19.840 --> 00:04:25.080
So, very quickly, like one of the applications that this enables that we've launched recently is live translation, right?

s55
00:04:25.080 --> 00:04:30.160
And this is a kind of application that kind of only works when you have all of these capabilities working within the same model.

s56
00:04:30.160 --> 00:04:32.720
You have basically state-of-the-art translation quality.

s57
00:04:32.720 --> 00:04:38.560
Even though this model is translating basically as the user or speakers are speaking, you kind of ask, "Hey, I I speak English.

s58
00:04:38.560 --> 00:04:42.120
There's maybe friends who are talking in Spanish and Italian and Chinese."

s59
00:04:42.120 --> 00:04:44.800
And it's translating all of them to your language as they talk.

s60
00:04:44.800 --> 00:04:52.320
Um and then we're we're finding is the translation quality for this like streaming real-time translation is about as good as you would get with offline systems, right?

s61
00:04:52.320 --> 00:04:55.720
Where you kind of know the full utterance um from the very beginning.

s62
00:04:55.720 --> 00:05:05.600
So, that's something that has been classically very hard to do with these cascaded systems, but with LLMs, it actually just a lot of it comes out of the pre-training.

s63
00:05:05.600 --> 00:05:09.919
So, of course, to do this task it needs to do multilingual switching because you could be translating across different languages.

s64
00:05:09.919 --> 00:05:11.680
You don't know what those languages are beforehand.

s65
00:05:11.680 --> 00:05:21.120
It needs to preserve the speaker voice of the source speaker and be able to understand multiple speakers, be robust to noise, and of course, like do all this in real time, right?

s66
00:05:21.120 --> 00:05:29.720
So, again, it would be very hard to try to engineer this, but then with the LLM and Gemini models, this almost becomes a prompting task.

s67
00:05:29.720 --> 00:05:38.280
And on that, like, I know, in this diagram we're saying, "Hey, at the top with these models, if you prompt it to do the speaking like this streaming translation task,

s68
00:05:38.280 --> 00:05:41.280
and you give it the audio, it will produce the streaming translation output, right?

s69
00:05:41.280 --> 00:05:50.040
The same model, if you ask it to act like an agent and respond to maybe image and audio input, maybe asking questions about that image, it will give you an answer.

s70
00:05:50.040 --> 00:06:02.640
And finally, like very well we'll show examples of this, you can also have it create this embodied, you know, virtual agent that has a face, that has, you know, things and tools that it can show you, and it will produce this sort of embodied agent experience.

s71
00:06:02.640 --> 00:06:10.200
So, with that, I'm going to give it to Valeria to talk more about the North Star and some of the key demo products that we built.

s72
00:06:10.200 --> 00:06:20.680
Yeah, so to create this type of kind of universal, versatile, uh kind of model/product, there's like three vectors that we think about when we do research and product for these models.

s73
00:06:20.680 --> 00:06:24.720
Um and they also come with some challenges, so I'll like walk you through some of them.

s74
00:06:24.720 --> 00:06:30.720
So, at the core of a speech-to-speech model, the first thing that people usually think about is that it has very conversational, right?

s75
00:06:30.720 --> 00:06:34.200
It's low latency, it's very conversational, very snappy, very natural.

s76
00:06:34.200 --> 00:06:38.560
But, I think within our team, we really don't only want this model to sound nice.

s77
00:06:38.560 --> 00:06:46.440
We also have two pillars at the top that we also really care about like pulling all together into one model, which is intelligence and it being multimodal.

s78
00:06:46.440 --> 00:06:57.320
So, when we talk about intelligence, we talk about, you know, task completion, instruction following, reasoning, like capabilities that the model needs to have natively in order to complete tasks and to like do things

s79
00:06:57.320 --> 00:07:00.080
uh that have high customer satisfaction, for example.

s80
00:07:00.080 --> 00:07:07.760
And on the other side of the Venn diagram, we also have the idea that these models should be very multimodal, both in audio in and audio out, right?

s81
00:07:07.760 --> 00:07:12.680
So, sometimes a user doesn't only want to input audio in and have that be the start of the conversation.

s82
00:07:12.680 --> 00:07:21.880
We need video, your screen sharing, PDFs, whatever you would want the model to interpret and understand, we should be able to stream it in and also produce output out of it.

s83
00:07:21.880 --> 00:07:26.080
So, that's kind of like the trifecta of which we think about speech-to-speech models.

s84
00:07:26.080 --> 00:07:28.520
Um and I want to add a caveat about ITNN.

s85
00:07:28.520 --> 00:07:32.240
I think actually the majority of our Gemini users are non-English speakers.

s86
00:07:32.240 --> 00:07:41.280
Um so, we put a big focus on having and making sure that all these capabilities work within not only, you know, EN-US, but all the languages that our customers care about.

s87
00:07:41.280 --> 00:07:52.840
Um of course, this also our North Star also becomes one of the biggest challenges in our research because once you move one of the knobs, it's very easy for the other knobs to kind of like mess up, right?

s88
00:07:52.840 --> 00:07:57.160
Like a very quick example, oh, how do we increase intelligence in the model?

s89
00:07:57.160 --> 00:08:08.560
Well, you can turn thinking high or like the thinking is high as possible to have the model think a lot before calling a tool or answering a question, which in eval's it does show that it does improve the model's intelligence.

s90
00:08:08.560 --> 00:08:10.760
But when what does that do to latency, right?

s91
00:08:10.760 --> 00:08:13.919
And time to first audio and the naturalness of the conversation?

s92
00:08:13.919 --> 00:08:23.800
So, within the Gemini team, we're really trying to push forward research initiatives that can kind of blend in the three of them without really sacrificing any of those by a lot.

s93
00:08:23.800 --> 00:08:32.080
Um, but in the meantime, we're going to show you some of the demos that we think are hinting at how our speech-to-speech model can combine all these three into

s94
00:08:32.080 --> 00:08:33.599
really cool application.

s95
00:08:33.599 --> 00:08:41.919
So, the first one is you kind of saw this as a preview, but our live model, as you know, uh, powers streaming translation that supports 70+ languages.

s96
00:08:41.919 --> 00:08:47.600
So, this is a little bit around the core of conversation quality and IT&N efforts that we have in the team.

s97
00:08:47.600 --> 00:08:56.920
I'll play a quick video on how this works on Google Meets to help two people or maybe multiple people that are speaking different languages still have a live conversation.

s98
00:08:57.000 --> 00:08:58.800
Oh.

s99
00:08:58.800 --> 00:08:59.440
One sec.

s100
00:08:59.440 --> 00:09:00.560
Okay.

s101
00:09:00.560 --> 00:09:04.000
Okay, let's turn on speech translation.

s102
00:09:04.920 --> 00:09:14.040
So, here the user can simply select the language that they want the translation to happen in and then the rest will be done in real time in multiple speakers kind of having a conversation back and forth.

s103
00:09:14.040 --> 00:09:18.440
So, I'll show you a snippet of this video of what happens.

s104
00:09:20.440 --> 00:09:22.080
It's great to see you both.

s105
00:09:22.080 --> 00:09:25.320
Cassie, how's the weather in Shanghai?

s106
00:09:25.320 --> 00:09:26.800
It's nice to meet you.

s107
00:09:26.800 --> 00:09:29.760
The weather here is really nice, sunny and bright.

s108
00:09:29.760 --> 00:09:33.880
I spent all day Saturday in the park with my family.

s109
00:09:35.360 --> 00:09:37.320
That sounds wonderful.

s110
00:09:37.320 --> 00:09:40.480
Anna, you mentioned last week that you were celebrating your birthday.

s111
00:09:40.480 --> 00:09:41.280
How was it?

s112
00:09:41.280 --> 00:09:44.240
Oh, that was fantastic.

s113
00:09:44.240 --> 00:09:47.600
I had dinner at my favorite restaurant with some friends.

s114
00:09:47.600 --> 00:09:51.400
If you visit Sweden, you must try this restaurant.

s115
00:09:51.400 --> 00:09:52.800
It was

s116
00:09:52.800 --> 00:09:54.520
So, um a lot of things happening.

s117
00:09:54.520 --> 00:09:56.880
Not only there's, you know, real-time translation.

s118
00:09:56.880 --> 00:09:58.080
Whoops.

s119
00:09:58.080 --> 00:10:00.760
How do I back to Okay.

s120
00:10:00.760 --> 00:10:15.520
Um not only there's real-time translation that is happening, uh but we see that it's in a multi-speaker setting, you know, low latency, like right after the user starts speaking, the translation kicks off, so that it doesn't feel like it's really turn by turn and robotic and you're afraid to interrupt because,

s121
00:10:15.520 --> 00:10:19.480
you know, once you start speaking, the model will start catching your translation directly.

s122
00:10:19.480 --> 00:10:26.280
So, now I'll show some other applications where we see that we have one single speech-to-speech model for many conversational frontiers.

s123
00:10:26.280 --> 00:10:33.360
So, two products that I want to highlight here are the same model that we power search live for everyday conversations in any language.

s124
00:10:33.360 --> 00:10:38.080
It's the same model that we use to power developer experiences in our live API.

s125
00:10:38.080 --> 00:10:49.240
As you'll see in the videos, both use cases are extremely different, but because we try and make this this model or this speech-to-speech model very promptable and universal, you can see how it adapts to the different use cases.

s126
00:10:49.240 --> 00:10:52.200
So, let me just play the first video.

s127
00:11:12.680 --> 00:11:14.920
So, to highlight in this video, many things are happening.

s128
00:11:14.920 --> 00:11:22.040
First, the multimodal, you know, it's real-time uh video and audio streaming in, so that you don't have to actually describe what the sofa looks like.

s129
00:11:22.040 --> 00:11:28.720
But then the multimodal out, you see that there was a tool called made to pull up the right search cards if the user wanted to know more information.

s130
00:11:28.720 --> 00:11:34.000
And then finally, the ITN component, you know, we're localizing the speaker's Spanish to Spain Spanish.

s131
00:11:34.000 --> 00:11:42.840
And also, when we do the kind of responding in Spanish, we're also smart enough to know that some words are better to be left in English, like the mid-century

s132
00:11:42.840 --> 00:11:43.360
type of word.

s133
00:11:43.360 --> 00:11:48.400
It didn't translate it directly to Spanish cuz it knew that the term was commonly known in English.

s134
00:11:48.400 --> 00:11:49.440
Sorry, in Spanish.

s135
00:11:49.440 --> 00:11:54.480
So, this is one example of how users are using search live for like info-seeking questions.

s136
00:11:54.480 --> 00:12:00.280
Now, let me show a very different demo on how we do this for conversational voice agents.

s137
00:12:02.400 --> 00:12:04.400
Oh.

s138
00:12:04.680 --> 00:12:06.680
Sorry.

s139
00:12:14.320 --> 00:12:18.560
Hi, I've blown a tire and had to pull over to the side of the road.

s140
00:12:18.560 --> 00:12:20.640
Can you help?

s141
00:12:20.640 --> 00:12:24.960
Hello, you've reached Auto Transform Insurance Roadside Assistance.

s142
00:12:24.960 --> 00:12:26.080
My name is Ace.

s143
00:12:26.080 --> 00:12:29.040
I understand this is stressful, but we'll take care of everything.

s144
00:12:29.040 --> 00:12:32.120
To get started, could I please have your name and your policy number?

s145
00:12:32.120 --> 00:12:34.480
Uh my name is Richard Jenkins.

s146
00:12:34.480 --> 00:12:38.560
I I don't actually have my policy number to hand.

s147
00:12:39.000 --> 00:12:40.280
No worries, Richard.

s148
00:12:40.280 --> 00:12:44.040
I can look you up with your registration plate and postcode instead.

s149
00:12:44.040 --> 00:12:46.440
Could you provide those for me, please?

s150
00:12:46.440 --> 00:12:48.520
Whoa, that was a bit close.

s151
00:12:48.520 --> 00:13:00.120
Um yeah, my registration plate is BD21 XYA and uh my postcode is SN48ZX.

s152
00:13:03.520 --> 00:13:07.839
Policy details BD21 XYA.

s153
00:13:07.839 --> 00:13:08.680
Thank you.

s154
00:13:08.680 --> 00:13:11.280
For your safety, please stay clear of the road.

s155
00:13:11.280 --> 00:13:15.600
I've found your details and I see you're in a blue Mini Cooper F-Series.

s156
00:13:15.600 --> 00:13:16.880
Is that the vehicle you're in?

s157
00:13:16.880 --> 00:13:19.760
Yeah, yeah, that that's the vehicle.

s158
00:13:20.440 --> 00:13:21.600
Got it.

s159
00:13:21.600 --> 00:13:26.320
Okay, so I'll pause it here, but as you can see, very different things are happening under the hood with the same model.

s160
00:13:26.320 --> 00:13:32.920
You know, it's more about alphanumeric accuracy for complex kind of like postcode numbers or addresses.

s161
00:13:32.920 --> 00:13:40.200
There's like this feature that we have called proactive audio, which is the idea that the LLM knows when or when not to respond to an external input.

s162
00:13:40.200 --> 00:13:50.680
So, for example, if you're in a conversation and someone else is talking or there's background noise happening in the back, the model knows to not stop or cut itself short because there's external noise happening.

s163
00:13:50.680 --> 00:13:57.320
Because we realized that the majority of these conversations are not happening in an office room with like no noise, you know, in a sealed environment.

s164
00:13:57.320 --> 00:14:00.480
They're happening on the go, on the train, while you're on a walk.

s165
00:14:00.480 --> 00:14:05.440
Um and that's the type of experiences that we want to facilitate with the speech-to-speech model.

s166
00:14:05.440 --> 00:14:11.400
Um lastly, um another thing we're really excited about is not only voice out, but also multimodal out.

s167
00:14:11.400 --> 00:14:16.960
We believe that, you know, the true AGI conversational frontier will also require visual presence.

s168
00:14:16.960 --> 00:14:24.680
So, we were excited to launch our first kind of pilot demo with City um in Cloud Next, which supports customized real-time avatars.

s169
00:14:24.680 --> 00:14:32.280
Um and you can kind of like personalize anything from a hyperrealistic human to a cartoon appearance and everything in the middle.

s170
00:14:32.280 --> 00:14:42.880
And powered by the same speech-to-speech model we've been showing, it allows to have, you know, low-latency, multilingual lip-syncing, and a really kind of like uh ongoing fluid conversation that has visual presence.

s171
00:14:42.880 --> 00:14:48.760
So, let me show you a demo that also brings our Venn diagram together and what we're excited about.

s172
00:14:48.760 --> 00:14:51.000
All this is reminding me of my daughter's college fund.

s173
00:14:51.000 --> 00:14:53.880
How are we tracking on that?

s174
00:14:55.000 --> 00:14:56.839
You're tracking well, Jackson.

s175
00:14:56.839 --> 00:15:01.120
I've also identified a new opportunity that may get you there even sooner.

s176
00:15:01.120 --> 00:15:03.640
Oh, and I can see Lisa just joined you.

s177
00:15:03.640 --> 00:15:04.680
Hi, Lisa.

s178
00:15:04.680 --> 00:15:07.320
She must be so excited about her college acceptance.

s179
00:15:07.320 --> 00:15:08.600
Congratulations.

s180
00:15:08.600 --> 00:15:11.880
It's wonderful to see your savings goals coming to life.

s181
00:15:11.880 --> 00:15:12.839
Excelente.

s182
00:15:12.839 --> 00:15:16.720
Las cosas han estado tan inestables últimamente.

s183
00:15:18.360 --> 00:15:19.120
Sí.

s184
00:15:19.120 --> 00:15:25.280
Los servicios de tecnología y comunicaciones han mostrado un desempeño sólido en lo que va del año.

s185
00:15:25.280 --> 00:15:26.560
That's good news.

s186
00:15:26.560 --> 00:15:36.400
I always joke that the user's audio in Spanish is worse than the audio model speaking back, but um this is kind of just to show how like our Venn diagram of combining, you know, multimodality

s187
00:15:36.400 --> 00:15:49.200
in and out, you know, tool calling to like pull up the relevant examples from the user, and also conversational fluidity with ITNN are starting slowly to come together in these types of demos that we're excited to keep pushing the frontier of.

s188
00:15:49.200 --> 00:15:57.520
Um so with this parting thought, I guess last kind of thought that we have for you is that we believe that AGI will not be typed, that it will be spoken.

s189
00:15:57.520 --> 00:16:08.680
Um and for it to be spoken, there's a lot of things that need to work together in a single promptable, versatile model that allows a user to switch between all the sorts of conversation modes that we're looking at, right?

s190
00:16:08.680 --> 00:16:17.480
From translation to taking action to brainstorming to rambling, and we truly believe in the power of these speech-to-speech models to achieve that like seamless switching.

s191
00:16:17.480 --> 00:16:19.760
Um so we're excited to push the frontier on that.

s192
00:16:19.760 --> 00:16:25.600
So if you're excited or want to learn more, please come talk to us, and thank you so much for coming.

s193
00:16:38.231 --> 00:16:40.231
[music]