Can Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff

AI Engineer · 20 min · 260 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Can you hear me?
2. 00:13Fantastic.
3. 00:14Hello everyone.
4. 00:16Well, hello couple people that wandered in.
5. 00:19Um, I'm sorry the schedule has been changing around.
6. 00:24I'm sorry you didn't find the quiet working area you were looking for.
7. 00:29All right.
8. 00:30Um, we are looking at some agents with Gemini API.
9. 00:36Now, before we kick off, let's have a little listen to this.
10. 00:41Welcome one and all to the grandest stage on Earth.
11. 00:46You're standing right at the heart of the largest AI engineer World's Fair in history.
12. 00:52Let's build the future together, shall we?
13. 00:56Does anyone know what model that is?
14. 01:01Yeah.
15. 01:02You you you you're not allowed.
16. 01:04Anyone else?
17. 01:05Anyone know which model?
18. 01:07No. Any guesses?
19. 01:11It starts with G. Gemini.
20. 01:18Yes, it is a Gemini model.
21. 01:20But which one?
22. 01:24No, it's not life.
23. 01:25That's incorrect.
24. 01:28Yes.
25. 01:28This is Gemini TTS.
26. 01:30Is that what you were about to say?
27. 01:32No. Okay.
28. 01:33Um, so Gemini text to speech.
29. 01:36Uh, actually a really incredible model for expressive speech.
30. 01:41You have inline audio text you can steer it with.
31. 01:44Um, you can stream the speech.
32. 01:46It's a lot lower latency now than it used to be.
33. 01:48So uh if you are in the market for some speech uh text to speech that is Gemini TTS is a good model.
34. 01:59Now let's have a look at uh some of the things we're doing today.
35. 02:04Uh first of all my name is Thor.
36. 02:07I work on the developer experience at Google DeepMind.
37. 02:11So that mostly includes Gemini API and Google AI Studio.
38. 02:16So if you're using the Gemini models through the API, uh you know, let me know if you have any feedback.
39. 02:23I have my colleague Philip here as well, uh who's working on managed agents on the API.
40. 02:28So if you're using that, uh let us know as well.
41. 02:32So this is a bit of a a prelook at the workshop we're doing tomorrow.
42. 02:38Um so tomorrow we'll do kind of Gemini API especially focused on um live agents uh conversational agents real time uh conversational agents multimodal real-time conversational agents we're just packing everything in there
43. 02:55um and so what we can look at actually there's uh been a lot of things that shipped and so specifically what we want to look at is the multimodality
44. 03:04of Gemini and the gen media models Uh so actually this slide is already out to date because this morning we shipped a new model uh that is called anyone know which model did we ship?
45. 03:17We shipped Omni Flash which is uh a video generation and editing model and we shipped Nano Banana Light.
46. 03:26So Nano Banana Light is a really exciting model because it cuts down the latency of image generation and image editing quite significantly.
47. 03:35And so what I've done, we'll we'll do a couple demos here.
48. 03:39Uh and one of the demos is um this example here, which I call the uh Gemini Nanobanana story canvas.
49. 03:48Uh and I just VIP code this up uh just recently and I'm having some trouble with my mouse here.
50. 03:55Um but so what we can do is we can come up with a story with um Gemini Life and then have it use Nano Banana Light to to illustrate that story for us.
51. 04:07So anyone you know this is a little bit like improv theater now.
52. 04:10Anyone has a story line they want to start with?
53. 04:15Where are we somewhere?
54. 04:18Okay, let's say um we're in Taipei in the 1980s.
55. 04:27The neon lights of Taipe's night market pulsed with an almost magical energy reflecting off the wet pavement and the chrome of passing scooters.
56. 04:36Rain had just passed, leaving a humid mist that hung in the air.
57. 04:40Amidst the throng of shoppers and vendors, a young woman clutched a small, strangely glowing object under her jacket.
58. 04:48Something very exciting happened in 1980s Taipei.
59. 04:51A new kind of drink emerged.
60. 04:54She paused at a stall where a man was vigorously shaking a metal cup, creating a frothy, milky concoction.
61. 05:01He handed a cup to a customer who took a long sip with wide eyes, savoring the strange tapioca pearls at the bottom.
62. 05:08This was the birth of bubble tea, and its sweet, chewy magic was spreading like wildfire.
63. 05:18So what you can see here is uh from the tool call.
64. 05:21So Gemini Life is getting a tool call um and then it's using the Gemini interactions API to generate an image on the fly.
65. 05:31Now it still takes kind of you know two to three seconds for the actual image generation but that is um pretty impressively fast.
66. 05:40And um what you can look at is if you're looking at the code here uh we're using the interactions API.
67. 05:46And so the interactions API is exciting because it is a stateful API which means we can chain our interactions very seamlessly.
68. 05:55So you could see here the first image generation that we were doing uh it didn't have a previous interaction ID but then the next generation that we were doing we're passing this previous interaction ID
69. 06:09to the interactions API which means that we have the previous context of the previous interaction and we can continue that in the generation.
70. 06:18So what that means specifically for our demo here is that you can see is our first you know interaction uh it generated an image with a very specific kind of visual feeling and also with
71. 06:30characters right and so we can continue that interaction and so we can keep the same consistency kind of across these generations.
72. 06:38So if you haven't tried out the interactions API, uh it is our unified API for interacting with all of the Google DeepMind models as well as agents.
73. 06:48Uh and so do try that out.
74. 06:50It is a it is a really really cool um way to interact with everything that Google DeepMind has.
75. 06:57Okay, so this was you know a bunch of things that happened.
76. 07:01Um so this was kind of on the Gen Media side.
77. 07:04We shipped uh Nano Banana Light this morning.
78. 07:07Now, we also recently shipped on the audio side our live translation model.
79. 07:12Has anyone tried out Gemini 3.5 live translate?
80. 07:18Yes.
81. 07:18Okay, a couple folks.
82. 07:19Uh, that's fantastic.
83. 07:21So, what we can do is uh we can try that out as well.
84. 07:24We can play uh a little demo.
85. 07:27And so what I'm going to do here is I'm going to create this live translation session and I want you to scan uh this QR code on your phone.
86. 07:39And so what this is doing here is if I now enable um my microphone.
87. 07:45So this is getting the the microphone of my speech um here into the session.
88. 07:51And now if you go in and you select your preferred language.
89. 07:56So we're seeing here we're getting some German.
90. 07:58So someone is listening to me now speaking in German.
91. 08:02And so if you are uh now listening to this, so you probably need some headphones.
92. 08:06So if you have some noiseancelling headphones, you can now listen into this talk in your preferred language.
93. 08:14And so Gemini live translate is a really incredible model.
94. 08:18It's near realtime translation.
95. 08:20So as I'm speaking, as soon as the model is recognizing kind of the context of what I'm speaking about, uh it will then start translating in real time to your desired language.
96. 08:33So I see someone is listening in there.
97. 08:35Do do you get some audio out?
98. 08:36Can you hear it?
99. 08:38Yeah.
100. 08:38Does it sound good?
101. 08:39Fantastic.
102. 08:40See, um, so this is this is really exciting because you know in a conference like this obviously we have a very very international audience and so I can speak in my preferred language.
103. 08:52So I could switch to German now for example.
104. 09:11the powerful Gemini live translates.
105. 09:14Um, it is a a really incredible model.
106. 09:17Um, it automatically detects the language the speaker is speaking and then translates it in real time uh into our preferred language.
107. 09:26Okay, we have uh a very international crowd here today.
108. 09:30Uh, who's listening in in uh Chinese?
109. 09:33Are you listening in Chinese?
110. 09:37Oh, okay.
111. 09:38I didn't understand that, but okay.
112. 09:44Um, so that is Gemini live translate.
113. 09:48Um, if you want to try it out, you can add it to your um applications now.
114. 09:54It is available on the Gemini API.
115. 09:56So, do try that out.
116. 09:58uh it is a really incredible model and uh would love to hear kind of what you're what you're building with that.
117. 10:04Now obviously we're looking at uh agents, right?
118. 10:09Can your agents uh what's it the title of this talk?
119. 10:12Gemini made up this title.
120. 10:14So I uh can your agent hear me now?
121. 10:16I think that that is the title.
122. 10:18So our most recent real time conversational model is called Gemini 3.1 Flash Life.
123. 10:26Now, if you've talked to Gemini before in the Gemini app, that is exactly the model that you're speaking to.
124. 10:34And so, we can do a little demo here.
125. 10:36So, this demo was vipcoded by Gemini in Google AI Studio.
126. 10:41And if you remember the good old times, you know, when you were listening to radio, you know, not too long ago, you could phone up the the the radio host and you could ask for a song, right?
127. 10:54You could wish for a song.
128. 10:55So, it's the same idea here.
129. 10:57We're using Gemini Life to be our radio DJ and then we're giving Gemini Life an API, a function call to the Gemini API to then use LIA 3, which is our music generation model
130. 11:11to then generate some music for us on demand.
131. 11:14Okay, let's uh try this out.
132. 11:17We're we're just going live demo to live demo here.
133. 11:20Let's let's see.
134. 11:22Well, hello there.
135. 11:23You're live on the live jukebox right now.
136. 11:25What absolute banger are we cooking up for you today, mate?
137. 11:28To get this track properly sorted, I need a bit more of an idea.
138. 11:32What genre are we thinking?
139. 11:33Upbeat synth pop, smooth jazz, something with a bit more well.
140. 11:37And what's the vibe lyrics-wise?
141. 11:38Give us some details.
142. 11:39Yeah.
143. 11:40Can we get like a really high energy ragaton, please?
144. 11:44About the AI engineering world's fair here in San Francisco.
145. 11:50Now that sounds like a proper belter, mate.
146. 11:52high energy regaton about the AI engineering world's fair.
147. 11:56Brilliant.
148. 11:56Right.
149. 11:56To make this absolutely top of the pops, any specific bits of tech or big names you want thrown in, or are we just going for that huge festival vibe?
150. 12:05Yeah, just surprise me, please.
151. 12:09You got it, my friend.
152. 12:10Prepare for a huge musical surprise generating that track right now.
153. 12:14Mixing that regaton rhythm with the tech buzz for the World's Fair.
154. 12:19It's going to blow your headphones off.
155. 12:21Fair.
156. 12:21What a feeling.
157. 12:22Let's go on the floor.
158. 12:25Neural networks firing up.
159. 12:26Groundbreaking for sure.
160. 12:28Clever algorithms dancing.
161. 12:30Tech pioneers mingling.
162. 12:31This is innovation.
163. 12:32Baseline is pulsing heavy.
164. 12:34Can you feel the heat?
165. 12:35Sin stabs electrifying.
166. 12:36Move in your feet.
167. 12:41This is how we are.
168. 12:50Yeah, that that was pretty cool.
169. 12:51You went multilingual there as well.
170. 12:54Nice one.
171. 12:54Cheers.
172. 12:57Cheers, mate.
173. 12:57Glad you like the flavor.
174. 12:59Keep those big ideas coming.
175. 13:00This jukebox never sleeps.
176. 13:02What are we spinning next then?
177. 13:03Feeling.
178. 13:04All right.
179. 13:05Um, yes, you can try this out yourself.
180. 13:08Uh, it is is a a demo that was built in Google AI Studio.
181. 13:13Uh so if you want you can fork this in AI studio and continue yourself.
182. 13:19Um I know this application might not necessarily be that useful for your business context, you know, if that's something you're looking to do, but I think the individual components
183. 13:30are definitely useful.
184. 13:31So if you're building real-time conversational agents, uh you know, you want them to be multilingual, you want them to have capabilities, uh tool calls.
185. 13:41So in this example, we were just giving the agent a tool call to generate music with LIA 3 uh on demand.
186. 13:49Now obviously this could also just call out to your CRM and get some you know customer information or it could create you know a new ticket on the back end.
187. 13:58Um so you know really what I want you to take away is kind of this real time multimodal conversational uh experience that you can build.
188. 14:07And so the way that you can build with this model, so the the Gemini 3.1 flash real-time conversational model is through the life API.
189. 14:18So the life API is a low latency real time voice and vision uh API.
190. 14:25So you can build multiodal uh life real-time interactions.
191. 14:30And so what this means is you have this stateful websocket API where you create a session and then over this websocket you can send in real time text, audio and video frames to the model
192. 14:45and the model will then respond in real time back in audio chunks.
193. 14:50And these audio chunks you can then stream uh in your application.
194. 14:54Uh and in addition to the audio, you also get the transcript of what was said.
195. 15:00So if you've worked with voice agents before, this is a little bit different from kind of your cascading voice agent pipeline because this is a native audio model.
196. 15:11So the model itself does not go through text, but rather the intelligence is baked into the audio model.
197. 15:19So the audio model can understand the speech naturally and natively and can then respond back in any of the 90 supported languages.
198. 15:30So this is what allows us to to switch very fluidly between different languages.
199. 15:36Um so kind of the key features here, yeah, multilingual support.
200. 15:40So 90 uh different languages currently supported in the preview.
201. 15:44Um we can barge in, we can interrupt the model, right?
202. 15:47If our DJ is talking a bit too lengthy, uh we we can we can barge in, interrupt it, we can give it uh the tools and we have the audio transcription as well.
203. 15:57Now, if you're building real time conversational applications or real-time um multimodal applications, uh you know, real-time media can be quite challenging to scale.
204. 16:09So, we have a great lineup of partners.
205. 16:11So, you know, LifeKit, Pipecat, they all actually here in the expo as well.
206. 16:16I recommend you go visit them.
207. 16:17They do some really really cool stuff to allow you to get up and running quickly with the Gemini Life API.
208. 16:25Okay, so if you're now um you know, you're like, "Okay, this sounds great.
209. 16:30I want this."
210. 16:31Um the easiest way to try this out is through Google AI Studio.
211. 16:36So we spend a lot of money on this domain, AI.studio.
212. 16:40So I do appreciate if you use that one.
213. 16:43Um so we can get some ROI there.
214. 16:45So ai.studioife.
215. 16:48You can try out the model.
216. 16:49You can talk to it and you can understand the capabilities that are available.
217. 16:54So let's do that now.
218. 16:56Uh so we're just going to AI.dev.
219. 16:59We also have AI.dev.
220. 17:01Um yeah, please do use that as well.
221. 17:03That one was was also not not so cheap.
222. 17:07Uh there we go.
223. 17:08AI.devlife.
224. 17:10And so here uh we can also try out the live translate model if you want to.
225. 17:15Uh but we can here select our Gemini 3.1 flash uh live model.
226. 17:21We can select a different voice.
227. 17:23Uh for example, I like puck here.
228. 17:25And then also we can enable grounding with Google search.
229. 17:29So obviously the model is trained on a lot of information but it does not have access to real time information, right?
230. 17:37And so by giving it access to Google search, it can actually go off and it's basically like a built-in function call.
231. 17:45And if we ask it about today's weather or you know why Germany played so well in the World Cup, uh we can we can do that and it can use Google to to look that up.
232. 17:56Okay, let's give this a try.
233. 17:58Hey, can you hear me?
234. 18:01Yes, I can hear you perfectly.
235. 18:03How can I help you today?
236. 18:04Can you see me?
237. 18:07Yes, I can see you clearly.
238. 18:08You're wearing a black shirt and a black hat sitting in front of a blue background.
239. 18:13Anything specific you wanted to show me?
240. 18:15Yeah.
241. 18:15What does it say on my sweatshirt?
242. 18:18I can see the word Gemini printed in yellow letters.
243. 18:21Anything else you'd like me to identify?
244. 18:44Uh, okay.
245. 18:45I think my uh my Mandarin didn't work out so well there.
246. 18:50Uh, can you also speak Mandarin, please?
247. 18:53Yes, I can speak Mandarin, too.
248. 18:55What would you like to say now?
249. 19:06What's about?
250. 19:13All right.
251. 19:14If you don't read Mandarin, I do apologize.
252. 19:16I was just asking the model what it has eaten, and I just said it doesn't need to eat because it's it's a model.
253. 19:23So, uh, yeah, there we go.
254. 19:27All right.
255. 19:28So this is the easiest way for you to get started with uh Gemini ai.studiolife.
256. 19:37If you want to learn more about building with Gemini and the Gemini live API, join me tomorrow morning in uh room 2009, I think.
257. 19:49Check the schedule.
258. 19:50It it does change every once in a while.
259. 19:52Uh but other than that, Dankur, thank you.
260. 19:56Cheers.