WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=oWTEiYpxl80

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/oWTEiYpxl80.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.639 --> 00:00:13.840
So, hi everyone.

s2
00:00:13.840 --> 00:00:17.039
Uh, welcome to why agents should have their own sandbox.

s3
00:00:17.039 --> 00:00:17.920
Uh, I'm Philip.

s4
00:00:17.920 --> 00:00:22.480
I'm part of the Google DeepMind team, mostly working on AI studio and Gemini API.

s5
00:00:22.480 --> 00:00:28.800
Uh, we have a small crew, so I try to keep the boring stuff short and like the exciting demo stuff long.

s6
00:00:28.800 --> 00:00:31.439
And we can also ask some questions I guess at the end.

s7
00:00:31.439 --> 00:00:35.200
Maybe you need to sit a little bit further at the front so I can understand you.

s8
00:00:35.200 --> 00:00:38.879
Um so I want to start with a little bit of a background.

s9
00:00:38.879 --> 00:00:55.039
I'm not sure if you have seen that slide but that's a very popular slide when you do like Google presentations um to show a little bit um what we have achieved in the last two year with like the Gemini models but also our open models with Gemma and the slide is already outdated since yesterday.

s10
00:00:55.039 --> 00:01:00.160
So yesterday we shipped Nano Banana Light uh which is our new image generation model.

s11
00:01:00.160 --> 00:01:05.280
Super fast uh sorry super fast uh super cheap.

s12
00:01:05.280 --> 00:01:11.680
Um it's like 4 cent per image and then also we shipped Omni flash on the interactions API.

s13
00:01:11.680 --> 00:01:20.080
So you can now programmatically use Omni to create videos to edit videos to yeah build gen media applications.

s14
00:01:20.080 --> 00:01:25.920
And to give a little bit of a brief history, I assume everyone has heard about agent has heard about large language models.

s15
00:01:25.920 --> 00:01:29.200
But to really quickly reflect where we are coming from.

s16
00:01:29.200 --> 00:01:37.920
So I think around 2018 19 with the first bird models and GBT models, we were super excited because we could continue sentences, right?

s17
00:01:37.920 --> 00:01:49.840
We were able to generate poems without uh with providing like a first few words and then the model coherently continue the text which was nice but not very useful or helpful for any one of us to

s18
00:01:49.840 --> 00:01:50.960
build something.

s19
00:01:50.960 --> 00:01:54.399
Then we we moved into the the instruction uh way.

s20
00:01:54.399 --> 00:02:02.399
Before we had pre-trained models which could continue text and then with instruction following we basically trained models to answer questions more or less.

s21
00:02:02.399 --> 00:02:06.560
So we told ask a question and the model correctly responded.

s22
00:02:06.560 --> 00:02:10.879
Um and then with with chat GPT um everything changed a little bit.

s23
00:02:10.879 --> 00:02:20.480
I think most of the people probably got started with that and we went into this user model turnbased conversation where we had a user input, a model input, a user input, a model output.

s24
00:02:20.480 --> 00:02:25.680
Then afterwards we combine that with function calling because we wanted the model to take actions for us.

s25
00:02:25.680 --> 00:02:31.599
And now we are in like this black box with goals, thinking and functions where we no longer just prompt the model.

s26
00:02:31.599 --> 00:02:44.239
We basically give a goal to the model, tell them how to verify its goal or how to like deploy it and like hopefully have it running for multiple hours or even days and come back with the results we want to achieve.

s27
00:02:44.239 --> 00:02:51.840
Um, so if you haven't used Gemini API or AI studio before, you can go to AI.studio or toi.dev.

s28
00:02:51.840 --> 00:02:54.000
So very quick, very short URLs.

s29
00:02:54.000 --> 00:02:55.519
H, get your own API key.

s30
00:02:55.519 --> 00:03:02.480
So if you have your computer with you and you if you want to follow along a little bit later on like trying out a few things we have shipped recently

s31
00:03:02.480 --> 00:03:03.200
you can do that.

s32
00:03:03.200 --> 00:03:06.640
Um no setup required can use your Gmail account.

s33
00:03:06.640 --> 00:03:08.959
Um can start start experimenting.

s34
00:03:08.959 --> 00:03:14.720
If you want to use bigger models more usage enter your credit card and we improved that a lot.

s35
00:03:14.720 --> 00:03:17.920
So you don't need to leave AI studio anymore to go into Google Cloud.

s36
00:03:17.920 --> 00:03:19.680
You can do everything inside of AI Studio.

s37
00:03:19.680 --> 00:03:23.680
You can set up your own budgets that nothing gets out of control.

s38
00:03:23.680 --> 00:03:31.920
And I want to start about or talk about the interactions API which is our new uh API for Gemini models and Gemini agents.

s39
00:03:31.920 --> 00:03:36.080
And on the right side sorry left side we have some code snippets.

s40
00:03:36.080 --> 00:03:44.000
So if you never have heard about the interactions API you can think a little bit about that's our answers to like the responses API from OpenI.

s41
00:03:44.000 --> 00:03:49.920
It is much more developer focused, much more simpler, much more web developer friendly.

s42
00:03:49.920 --> 00:04:00.080
And you can already tell in like the two snippets here uh to call a model like Gemini or to call an agent here like the anti-gravity agent, it's like literally the same interface.

s43
00:04:00.080 --> 00:04:03.280
And this works for all of the modalities Gemini offers.

s44
00:04:03.280 --> 00:04:06.239
So you have a model or an agent, you have an input.

s45
00:04:06.239 --> 00:04:10.560
The input can be a basic string or it can be multimodal data.

s46
00:04:10.560 --> 00:04:20.320
And then you have tools or environment which uh I'm very excited and why which we are going to talk about later why every model or agent should have its own box.

s47
00:04:20.320 --> 00:04:23.600
Uh new with the interactions API is you have serverside state management.

s48
00:04:23.600 --> 00:04:28.880
So if you don't want to manage the state on the client you can hand it over to the server.

s49
00:04:28.880 --> 00:04:30.479
We have long running operations.

s50
00:04:30.479 --> 00:04:33.680
So async requests basically you start something.

s51
00:04:33.680 --> 00:04:40.479
If we talk about omni video generation takes quite a long time and you might not want to keep your HTTP connection open.

s52
00:04:40.479 --> 00:04:52.000
So you send a request get back an ID can set up streaming or polling to wait uh until the the generation is completed and then um it comes with a lot of simplifications

s53
00:04:52.000 --> 00:04:53.759
for building agents and tool use.

s54
00:04:53.759 --> 00:05:00.160
So you can combine built-in tools like Google search uh with custom function calling and to look at some code examples.

s55
00:05:00.160 --> 00:05:08.720
So for serverside state it looks very similar if you have used the responses API we call it interactions because we just not get a response anymore.

s56
00:05:08.720 --> 00:05:10.639
We interact with models and agents.

s57
00:05:10.639 --> 00:05:19.919
You send an input and if you want to do a follow-up turn you provide a new input and basically just the ID from the previous turn and then on the server we

s58
00:05:19.919 --> 00:05:27.440
um concatenate all of like the different inputs and make sure that the model really reh sees what has been done so far.

s59
00:05:27.440 --> 00:05:29.600
And this goes for all modalities.

s60
00:05:29.600 --> 00:05:32.639
So for audio generation you have a model.

s61
00:05:32.639 --> 00:05:34.400
We have our Gemini TTS model.

s62
00:05:34.400 --> 00:05:47.840
We have our input and since we want audio back not text you define a response modality with audio and some generation configuration which was you for example want to use the same pattern works for image generation.

s63
00:05:47.840 --> 00:06:03.680
So our image generation model uh nano banana here our input some generation uh generation config and then you get back your image and for tool use uh also much better I would say than previously.

s64
00:06:03.680 --> 00:06:13.520
You just define your tools with like Google search URL context or your own custom function and then you send your request and the model decides on it on its own.

s65
00:06:13.520 --> 00:06:23.199
do I first need to like run Google search or call the function and all of the code you see here is basically enough to create a very simple small CVE agent

s66
00:06:23.199 --> 00:06:39.280
to detect if there are like any CVE um for react in this case and then like it uses Google search it uses URL context to visit those websites and then function calling to basically give structure to it so you can like save it in your database or look it up for example

s67
00:06:39.280 --> 00:06:49.520
um and something new and maybe different to other model providers is that we moved away from this user model term principle.

s68
00:06:49.520 --> 00:07:00.080
So if you build an application with another LLM provider or with open models, you have always this uh user model user model turn concept.

s69
00:07:00.080 --> 00:07:04.560
So you as a user provide an input and expect the model to return an output.

s70
00:07:04.560 --> 00:07:10.720
This worked until like one or two years ago I would say where we just got back responses from those models.

s71
00:07:10.720 --> 00:07:14.560
But with agents and with reasoning, all of that changed, right?

s72
00:07:14.560 --> 00:07:21.599
Models now think for a very long time where they might expose some thoughts directly or might just expose a signature.

s73
00:07:21.599 --> 00:07:26.080
Function calling is not very uh a user role, right?

s74
00:07:26.080 --> 00:07:39.120
With normal or like with previous APIs, you would need to use the user turn to return some function output, which for me always felt a little bit weird because the output or the new input is coming from the environment and not from the user.

s75
00:07:39.120 --> 00:07:59.680
So that's why we moved away from like those turns into steps or like to um like synchronous timeline of steps where every action or generation is its own type which makes it very easy to extend to new features and capabilities which might come in the future but also makes it still like very simple to use because you

s76
00:07:59.680 --> 00:08:02.000
really define what it is done.

s77
00:08:02.000 --> 00:08:12.560
So if you build a function calling agent, you return a function result and not like a a user role user with some content which might be a function result or not.

s78
00:08:12.560 --> 00:08:18.639
And at Google IO we ship the anti-gravity agent uh which comes with its own sandbox.

s79
00:08:18.639 --> 00:08:26.639
So the anti-gravity remote agent inside the Gemini API is powered by the same agent harness which is used with the anti-gravity product.

s80
00:08:26.639 --> 00:08:33.760
So if you have used the IDE or the new anti-gravity 2.0 O or the CLI, you might have used that harness already.

s81
00:08:33.760 --> 00:08:34.880
Very important here.

s82
00:08:34.880 --> 00:08:38.080
When we talk about harness, we don't mean the same agent.

s83
00:08:38.080 --> 00:08:43.760
The harness is basically the the back end of our agent, but it might be different between different products.

s84
00:08:43.760 --> 00:08:46.480
So the anti-gravity IDE is a coding agent.

s85
00:08:46.480 --> 00:08:55.600
So it might have a little bit of different tools or a little bit of of a different system instruction than the anti-gravity agent we have in the Gemini API, but it's still the same harness.

s86
00:08:55.600 --> 00:09:05.120
Meaning if we improve Gemini to become better with that harness, we benefit inside anti-gravity, but we also benefit inside of the Gemini API.

s87
00:09:05.120 --> 00:09:10.240
And with that new agent, we also introduce this uh environment field.

s88
00:09:10.240 --> 00:09:20.560
And an environment here remote is basically for that API call, your agent gets a sandbox managed by us.

s89
00:09:20.560 --> 00:09:35.440
Meaning if you send a request, our back end starts a cloud sandbox where the agent can run code, can create files, write files, um, install dependencies, install your own tools or maybe call your own APIs

s90
00:09:35.440 --> 00:09:40.880
and it's all managed and stateless on our side, meaning that you just send the API request.

s91
00:09:40.880 --> 00:09:45.200
You don't need to manage the infrastructure and those environments stay persistent.

s92
00:09:45.200 --> 00:09:58.880
So if you have multi-turn conversation where the agent um you set user input I don't know like um research the latest tech news and create a file and then you want to do like a follow-up interaction

s93
00:09:58.880 --> 00:10:07.680
you can like use the same environment from your previous turn with that new interaction but also this means that you can share environments between different agents.

s94
00:10:07.680 --> 00:10:22.079
So if you have like three agents working inside the same environment, they can share context through the file system or you can like even use sub agents to basically pick up a work uh another agent has been done and then of course environments

s95
00:10:22.079 --> 00:10:29.040
are very good but the model needs the right context and the right information or the right tools to work inside those environments.

s96
00:10:29.040 --> 00:10:35.839
So you can provide sources uh sources can be um GCS buckets, GitHub repositories or inline files.

s97
00:10:35.839 --> 00:10:38.560
It's very useful if you want to integrate skills.

s98
00:10:38.560 --> 00:10:44.079
So the anti-gravity agent is skills or file system native I would say.

s99
00:10:44.079 --> 00:10:52.240
So if you store skills inside the skills directory, it will automatically picked up and uh be pro uh included into the system instruction.

s100
00:10:52.240 --> 00:11:00.959
But also if you have an agents MD file from your coding agent or from your existing agents, it will also be automatically appended to the system instruction.

s101
00:11:00.959 --> 00:11:14.480
and then um you can create an agent because like sending a request is nice and having all of the configuration stateless is also very good but what if you want to share it or what if team one builds an agent team two wants to use

s102
00:11:14.480 --> 00:11:24.240
um therefore we created the agents API where you can define the base agent so what's the underlying brain so to speak you can define instruction and then also the

s103
00:11:24.240 --> 00:11:32.320
uh environment um and the environment can be a raw environment like with sources or you and uh use an existing environment.

s104
00:11:32.320 --> 00:11:42.880
So you can think about I have some kind of a scaffolding to do where I need to install like the GitHub CLI uh add some skills and maybe do some like sanity checks.

s105
00:11:42.880 --> 00:11:46.640
I can use the agent to create its own environment.

s106
00:11:46.640 --> 00:11:49.920
And after the agent is done, I get back an environment ID.

s107
00:11:49.920 --> 00:11:54.160
I can use that environment ID to create the new agent.

s108
00:11:54.160 --> 00:12:02.399
And every time I call this new agent, it will start from that already pre-loaded, preconfigured environment.

s109
00:12:02.399 --> 00:12:12.639
And so it's like very easy for you to use agents to build their own environments and then like to create agents out of them which you can share with your users or with your customers

s110
00:12:12.639 --> 00:12:20.959
without writing any Terraform without thinking about Kubernates, microservices, firecracker or anything else.

s111
00:12:20.959 --> 00:12:26.720
And to make it easier for you and for your coding agents, we created a CLI for the Gemini API.

s112
00:12:26.720 --> 00:12:28.720
So it's it's not a coding agent.

s113
00:12:28.720 --> 00:12:30.160
It's not Gemini CLI.

s114
00:12:30.160 --> 00:12:36.000
It's really an a CLI that your agent can use to call the Gemini API.

s115
00:12:36.000 --> 00:12:40.160
Meaning it can call Nano Banana or Omni if you want to generate some assets.

s116
00:12:40.160 --> 00:12:45.920
But you can also use it to create those agents, iterate on those agents, and improve those agents.

s117
00:12:45.920 --> 00:13:01.120
The QR code should bring you to like the GitHub repository and then also we have dedicated skills for working with the interactions API and the agents API um if you want to to play around with it later.

s118
00:13:01.120 --> 00:13:05.920
And now let's let's do some some demos.

s119
00:13:06.079 --> 00:13:11.120
So I'm in AI Studio inside the AI talk radio.

s120
00:13:11.120 --> 00:13:18.240
If you have seen or attended the Google IO developer keynote, you might have seen uh this already.

s121
00:13:18.240 --> 00:13:22.560
Um that's just a simple UI on top of managed agents.

s122
00:13:22.560 --> 00:13:23.600
We manage agent.

s123
00:13:23.600 --> 00:13:32.720
We are going to look at the agent in a little bit, but it's basically a talk show radio and that talk show u agent has access to all of the different Gemini models.

s124
00:13:32.720 --> 00:13:45.360
So it can generate a thumbnail, it can generate um speech, it can generate music, it can do research and it's basically a one short radio generator and we give it a try.

s125
00:13:45.360 --> 00:13:57.279
And I want a talk show radio on whether pineapples belong on a pizza or not.

s126
00:13:57.279 --> 00:13:58.480
And it's free to use.

s127
00:13:58.480 --> 00:14:16.000
So if you go to the applet later uh you can like have free daily usage to create create your own um AI talk show radio and what happens is like first step we start our environment and then the model starts thinking on what it needs to do and then the first step is like it installs our Python

s128
00:14:16.000 --> 00:14:20.639
dependencies and it reads our skills and then it continues to work.

s129
00:14:20.639 --> 00:14:29.920
Since this will take a little bit, I already ran a talk show radio on whether we should put in cereal first or milk first.

s130
00:14:29.920 --> 00:14:34.000
And the very interesting part about a talk show radio, it's not like a podcast you listen to.

s131
00:14:34.000 --> 00:14:43.360
It's really you have multiple um people like reasoning and like sharing their opinions on it and we give it a quick listen.

s132
00:14:43.360 --> 00:14:50.810
Is there a correct physical order to preparing your breakfast bowl or are we flirting with culinary chaos?

s133
00:14:50.810 --> 00:14:51.839
[music]

s134
00:14:51.839 --> 00:14:54.399
Welcome to AI Talk Radio.

s135
00:14:54.399 --> 00:14:57.839
I'm Paul broadcasting from our London studio.

s136
00:14:57.839 --> 00:15:01.360
Today we are exploring the great breakfast debate.

s137
00:15:01.360 --> 00:15:04.079
Cereal first versus milk first.

s138
00:15:04.079 --> 00:15:08.399
We will objectively break down the pros and cons of both methods.

s139
00:15:08.399 --> 00:15:09.760
Let's go to our lines.

s140
00:15:09.760 --> 00:15:12.160
We have David calling from Boston.

s141
00:15:12.160 --> 00:15:14.560
David, you're on the air.

s142
00:15:14.560 --> 00:15:16.000
Yeah, thanks Paul.

s143
00:15:16.000 --> 00:15:18.720
Look, it's got to be cereal first.

s144
00:15:18.720 --> 00:15:20.480
It's basic logic, right?

s145
00:15:20.480 --> 00:15:23.279
Cereal is the independent variable.

s146
00:15:23.279 --> 00:15:27.680
You pour it, you see the volume, then you add milk to compliment it.

s147
00:15:27.680 --> 00:15:32.560
Plus, like if you pour dry cereal into a bowl of milk, it splashes everywhere.

s148
00:15:32.560 --> 00:15:34.160
It's a total mess.

s149
00:15:34.160 --> 00:15:41.040
And the mashed potato analogy, you wouldn't put gravy in a bowl and drop a lump of potatoes on top, right?

s150
00:15:41.040 --> 00:15:43.279
A striking analogy, David.

s151
00:15:43.279 --> 00:15:47.199
Cleanliness and portion control are certainly strong arguments.

s152
00:15:47.199 --> 00:15:48.959
Let's hear the counterpoint.

s153
00:15:48.959 --> 00:15:51.600
We have Sarah on the line from Sydney.

s154
00:15:51.600 --> 00:15:53.199
Sarah, welcome.

s155
00:15:53.199 --> 00:15:54.480
Hi, Paul.

s156
00:15:54.480 --> 00:15:56.800
Okay, look, David has it all wrong.

s157
00:15:56.800 --> 00:16:00.720
Milk first is it's a total game changer for texture.

s158
00:16:00.720 --> 00:16:01.519
Okay, cool.

s159
00:16:01.519 --> 00:16:07.120
So, if you want to play around with it yourself, you can go to AI.studio.

s160
00:16:07.199 --> 00:16:08.240
And on AI.

s161
00:16:08.240 --> 00:16:25.120
studio you have that build section on the left side and there's a gallery and gallery is basically pre-built applets use cases for Gemini and if you scroll down to it uh you have new build experience with and there's the talk show radio if you click on it you will find the applet

s162
00:16:25.120 --> 00:16:34.399
you can uh remix it meaning you can like adjust it so if you don't like talk show radios and you would rather have I don't know some real podcast

s163
00:16:34.399 --> 00:16:39.120
uh application you can like very easily together with Gemini adjust it to their own needs.

s164
00:16:39.120 --> 00:16:47.360
And since applets are all open uh in terms of code, we can also take a quick look on like how our agent really is.

s165
00:16:47.360 --> 00:16:50.079
And our agent is just files.

s166
00:16:50.079 --> 00:16:53.600
So we have an agents MD file inside our environment.

s167
00:16:53.600 --> 00:16:59.440
That file will be loaded on environment startup which is then included into the system instruction.

s168
00:16:59.440 --> 00:17:01.360
Very basic agent file.

s169
00:17:01.360 --> 00:17:04.240
It has all of like the instructions for our agent to do.

s170
00:17:04.240 --> 00:17:12.720
So it should do some research, write some scripts for the different people, generate music, generate a speech, mix everything together.

s171
00:17:12.720 --> 00:17:15.439
And to do all of those works, we have skills for it.

s172
00:17:15.439 --> 00:17:19.839
So if we look into like the TDS generation, we have a skill on how to run it.

s173
00:17:19.839 --> 00:17:21.360
We have a Python file.

s174
00:17:21.360 --> 00:17:26.000
That Python file itself allows the Gemini agent to call other Gemini models.

s175
00:17:26.000 --> 00:17:31.760
So it uses the interactions API to generate some speech which we can then use.

s176
00:17:31.760 --> 00:17:34.960
And that that's all it takes for our agent.

s177
00:17:34.960 --> 00:17:43.679
And if you want to take that agent from the UI to like a back end, it's really you just need to copy that file and create the agent and then use it.

s178
00:17:43.679 --> 00:17:53.200
And if you want to play around with the uh agent itself uh inside the playground, we have uh inside the model pickers a new agent section.

s179
00:17:53.200 --> 00:17:56.480
So if you want to test deep research, go test deep research.

s180
00:17:56.480 --> 00:17:58.880
But there's also the anti-gravity agent.

s181
00:17:58.880 --> 00:18:06.880
And on the anti-gravity agent, we have six different uh agent templates uh which you can like very easily test, select and and see how it works.

s182
00:18:06.880 --> 00:18:08.799
We go into like the the standard one.

s183
00:18:08.799 --> 00:18:15.919
And what I really like to do is like explore the environment like let's see what the agent finds out about where it is running.

s184
00:18:15.919 --> 00:18:24.720
We send our prompt and now it starts the environment should take for the first request like 7 to 10 seconds and then we see the agent like doing its work.

s185
00:18:24.720 --> 00:18:27.919
So we have some reasoning, we have some bash commands.

s186
00:18:27.919 --> 00:18:38.480
So it runs so it checks what uh version west version I'm running on what CPU memory I have and then we get back a response from our agent that is runs on Ubuntu

s187
00:18:38.480 --> 00:18:53.039
um has 8 vCPU 16 GB of memory python installed pip installed and then you can like really continue let's see um do you have do we have node installed

s188
00:18:53.039 --> 00:19:03.520
um so you can like very easily follow up um and it just continues based on like my prompt and then also you have the option on the right side to

s189
00:19:03.520 --> 00:19:04.559
download the environment.

s190
00:19:04.559 --> 00:19:12.960
So if your agent has created some files you want to extract out of the environment, share it, can very easily download it or there's an API for it as well.

s191
00:19:12.960 --> 00:19:16.480
And then um uh what else?

s192
00:19:16.480 --> 00:19:21.840
Uh to quickly show you again how it works, I can make that a little bit bigger.

s193
00:19:21.840 --> 00:19:34.640
So when you when we send a request um we like start the sandbox and it loads all of our files and then we have like this agent loop completely on our server.

s194
00:19:34.640 --> 00:19:47.120
So when you build agents normally a lot of times you need to manage the function calling on your client right you need to passse the function call run it some somewhere and then send back the function result and with the managed agents it's really

s195
00:19:47.120 --> 00:20:00.320
just a single API call and everything runs inside that remote sandbox you can also connect remote MCP servers or run uh local MCP servers inside of that environment and

s196
00:20:00.320 --> 00:20:11.360
um yeah really get started so if you are interested about it uh we have great documentation inside the Gemini API under agents with a quick start which walks you through how to send your first request,

s197
00:20:11.360 --> 00:20:21.120
build your first agent and get started and then of course uh install the coding skills to have claude codeex or whatever model you use to to help you get started.

s198
00:20:21.120 --> 00:20:22.720
Uh thank you for for coming.

s199
00:20:22.720 --> 00:20:26.160
If you have some questions I'm like here