WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=8aVbXXvJUY4

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/8aVbXXvJUY4.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.320 --> 00:00:13.599
Hey guys, my name is Ivan.

s2
00:00:13.599 --> 00:00:16.160
I'm on the developer experience team at Google Demine.

s3
00:00:16.160 --> 00:00:20.800
And today we're covering the brand new interactions API and our new offering manage agents.

s4
00:00:20.800 --> 00:00:25.359
And so to take a quote from a very famous paper, an interaction is all you need.

s5
00:00:25.359 --> 00:00:35.520
Before we move on to what the interactions API is and what the new manage agents offering is let's take a trip down memory lane and let's kind of trace how models have changed and how we've worked with models.

s6
00:00:35.520 --> 00:00:44.719
So when we first started working with models a lot of times you would have a single interaction you would send a message tell me a joke and then you might get a response for example

s7
00:00:44.719 --> 00:00:48.160
why don't scientists trust atoms because they make up everything.

s8
00:00:48.160 --> 00:00:54.000
After a while as models got more and more capable and they understood more about the world this became a bit more complicated.

s9
00:00:54.000 --> 00:00:59.840
We wanted to take these models, put them in different applications, but to do so, we needed them to be reliable.

s10
00:00:59.840 --> 00:01:06.320
And so to do so, we invented function calling, where models could create individual JSON objects with predictable structures.

s11
00:01:06.320 --> 00:01:14.159
Think like how when you want to register for a website, your web page sends a JSON payload to the back end with a username and a password.

s12
00:01:14.159 --> 00:01:20.400
And that enables the backend server to say, okay, we have a new user and now we can actually create a new account.

s13
00:01:20.400 --> 00:01:28.560
that made it possible for us to actually use models in different parts of this application whether it's extracting user information, parsing data and more.

s14
00:01:28.560 --> 00:01:32.560
But right now we're seeing this incredible cups where models are getting more capable.

s15
00:01:32.560 --> 00:01:38.320
We're seeing models being able to reason go for very extended periods of time before they come back to people.

s16
00:01:38.320 --> 00:01:45.840
And so you're moving from a place where we went from completions instruction and functions where we might ask, hey, what's the weather in New York City?

s17
00:01:45.840 --> 00:01:54.159
and the model has a call weather tool that it can get that data, ingest it, and generate a response to now a case where models have multiple tools under their belt.

s18
00:01:54.159 --> 00:02:02.960
They're able to call tools, think and reflect on what the response meant, interact with the environment, and ultimately generate a response.

s19
00:02:02.960 --> 00:02:06.320
And so this just brings us back to what is an agent.

s20
00:02:06.320 --> 00:02:11.760
Fundamentally, if you look at what an agent is, it's really a model that powers a huge chunk of it.

s21
00:02:11.760 --> 00:02:15.760
A lot of people have reduced this to just a language model running in a loop.

s22
00:02:15.760 --> 00:02:18.000
And for most cases, this is true.

s23
00:02:18.000 --> 00:02:25.040
If you have a really capable model, you'll find that as we've got models that become more and more capable, a lot of the scaffolding has fallen away.

s24
00:02:25.040 --> 00:02:36.640
A lot of models, for example, the latest Opus models or fable models are now just immediately just using a bash tool instead of individually specialized tools like a edit file or a read file tool like we used to do.

s25
00:02:36.640 --> 00:02:48.080
And so really what we have is a language model that's used to understand the context, the user's intent that's given a bunch of different functions, all different resources that it then interacts with its environment in.

s26
00:02:48.080 --> 00:02:55.440
It has some short-term memory and some long-term memory to keep track of what it's encountered and then an environment to execute functions in.

s27
00:02:55.440 --> 00:03:02.879
If you're building a coding agent, but you don't give it an environment where it can actually execute code, for example, it's not going to be able to do anything.

s28
00:03:02.879 --> 00:03:08.000
And so if you'd like to try any of our models, the easiest way to do so is actually with AI Studio.

s29
00:03:08.000 --> 00:03:12.879
And so I actually like to point out that my speaker notes today were actually coded on AI Studio.

s30
00:03:12.879 --> 00:03:15.120
I vived it out 2 minutes before this talk.

s31
00:03:15.120 --> 00:03:19.519
Uh you'll see the um you can see the conversation history over here.

s32
00:03:19.519 --> 00:03:27.040
Um I literally said I need a mobile responsive website that I can pass in speaker notes because I found out I wasn't going to be able to use presenter view.

s33
00:03:27.040 --> 00:03:37.840
And so what I did, I just gave it some brief instructions and before you know it, you have a simple application that you can use without any hassle deployed on a website that's accessible to all.

s34
00:03:37.840 --> 00:03:41.840
So we have a really generous uh free tier for most of my presentations.

s35
00:03:41.840 --> 00:03:43.360
I've just been using the free tier.

s36
00:03:43.360 --> 00:03:46.560
I haven't been using my Google unlimited tokens thankfully.

s37
00:03:46.560 --> 00:03:48.720
And I think it's the best way for you to try our models.

s38
00:03:48.720 --> 00:04:04.560
Whether it's our new speechtospech translation models where you take in some you speak in let's say English and it can translate to any language or you know you want to try our latest models like the Gemini 3.5 flash etc. It makes it very easy for you to do so with simplified UI toggles to just be able to

s39
00:04:04.560 --> 00:04:06.879
tune all of the different parameters.

s40
00:04:06.879 --> 00:04:08.720
So let's talk about the interactions API.

s41
00:04:08.720 --> 00:04:21.040
Now one of the things that we've seen especially as we've moved from models to a dynamic where now we have models and agents is that the kind of workloads that we want to support have changed significantly

s42
00:04:21.040 --> 00:04:32.960
right through the Gemini API you have nano banana you have you know Gemini 3.5 flash but you also have very capable agents like the deep research agent that's going to run for anywhere from three minutes and more

s43
00:04:32.960 --> 00:04:39.680
you interact with it you have some sort of research plan it's going to go out and do its own thing and it comes like a very deep and detailed

s44
00:04:39.680 --> 00:04:41.120
research reports.

s45
00:04:41.120 --> 00:04:48.720
Now, it's pretty difficult then to think about consuming models and agents through a whole bunch of these different endpoints.

s46
00:04:48.720 --> 00:04:55.199
Additionally, for a lot of the older kind of API endpoints, a lot of the data was in in very deeply nested objects.

s47
00:04:55.199 --> 00:05:00.240
And so you can imagine the kind of difficulties that we had ourselves when we were building in first-party integrations.

s48
00:05:00.240 --> 00:05:10.479
And we wanted these use cases where you might be able to generate a nanobanana image, do some research, generate a vo image from there or spawn out all these different agentic

s49
00:05:10.479 --> 00:05:12.960
kind of like agents that go out and do their own thing.

s50
00:05:12.960 --> 00:05:22.400
And so the interactions API is our sort of solution to this as we look to build an API that can support all of these different use cases and ones that we can build on ourselves.

s51
00:05:22.400 --> 00:05:27.840
you look to bring Gemini across the entire surface of the Google ecosystem.

s52
00:05:28.400 --> 00:05:32.240
So the best place to get started is just with serverside state.

s53
00:05:32.240 --> 00:05:42.720
One of the things that you'll realize especially as you use the new Gemini series of models is that we essentially have this thing called thought signatures where if you interact with the model, if it calls a function, if it gives a response,

s54
00:05:42.720 --> 00:05:46.160
it's going to pass back a very opaque series of these numbers.

s55
00:05:46.160 --> 00:05:56.400
For a lot of people, manually managing this was very difficult and sometimes when we talked to different startups, we found that they would lose their cache just with a single whites space that they mistakenly added in.

s56
00:05:56.400 --> 00:06:00.960
So now with the responses API, you can see on the first turn, hi, my name is Phil.

s57
00:06:00.960 --> 00:06:03.840
And in the response, we give you a interaction ID.

s58
00:06:03.840 --> 00:06:13.360
And as long as you give us back that same interaction ID under the parameter previous interaction ID that you'll see in the second turn, all that context is preserved.

s59
00:06:13.360 --> 00:06:21.440
And this is really important with the new Gemini series of models because if you don't give us back those thought signatures, you're going to see a decrease in performance.

s60
00:06:21.440 --> 00:06:23.360
But what else does this unlock?

s61
00:06:23.360 --> 00:06:35.120
Here you see a demo built by Vampsy from our team that did an incredible demo where using a single photo, you could generate with Nanovana all these different variations of yourself in different countries and different locations.

s62
00:06:35.120 --> 00:06:39.919
Each of these individual photos are a single interaction, but you can think of it from an initial API call.

s63
00:06:39.919 --> 00:06:42.319
It spawns all these different interaction calls.

s64
00:06:42.319 --> 00:06:50.240
Using those interactions, the same interaction ID, we then generate a video using the interaction ID and the new Omni model that we just launched.

s65
00:06:50.240 --> 00:06:55.440
Something like this would not have been possible without the interactions API built by Philip over there.

s66
00:06:55.440 --> 00:07:01.120
So, actually, it's a really incredible piece of work, I will say.

s67
00:07:01.120 --> 00:07:06.880
So, to give you an example, an idea of what it would take to build something similar, this is all it would take.

s68
00:07:06.880 --> 00:07:11.759
First, you pass in Gemini 3.1 flashlight image, generate an image of me in Venice.

s69
00:07:11.759 --> 00:07:21.759
You take the interaction ID, you chunk it back into the client.create, and this time you specify that you want the Omni flash model instead that we launched on Wednesday.

s70
00:07:21.759 --> 00:07:29.280
And that allows you to now use the same Omni model with the same information, the same kind of images, and the same context throughout all these calls.

s71
00:07:29.280 --> 00:07:36.720
It's making it possible to build more complex, rich, and just these really complex agentic applications.

s72
00:07:36.720 --> 00:07:38.880
The same thing occurs with audio generation.

s73
00:07:38.880 --> 00:07:42.400
If you look at the bottom, you see that now we have output.type.

s74
00:07:42.400 --> 00:07:48.560
In the past, you would have to take this very nested object like completions audio.

s75
00:07:48.560 --> 00:07:50.479
And it would be very tricky for you to nest it out.

s76
00:07:50.479 --> 00:07:52.800
I can't even remember it off the top of my head.

s77
00:07:52.800 --> 00:07:57.440
But now, every single bit of the outputs that come out are clearly demarcated by a strong type.

s78
00:07:57.440 --> 00:08:02.560
If it's an audio, as you see over here, all you got to do is just parse it and handle it accordingly.

s79
00:08:02.560 --> 00:08:03.039
Right?

s80
00:08:03.039 --> 00:08:05.759
But what happens if now I want to generate an image?

s81
00:08:05.759 --> 00:08:09.199
It's the same method over there, client.interaction.create.

s82
00:08:09.520 --> 00:08:14.720
All I got to do is just modify the response modality and the generation config.

s83
00:08:14.720 --> 00:08:16.879
And now I have an output.type of image.

s84
00:08:16.879 --> 00:08:21.680
And this makes it very easy for me to start building and experimenting with Gemini.

s85
00:08:21.680 --> 00:08:23.599
The same thing comes with tool use.

s86
00:08:23.599 --> 00:08:26.720
You can mix and match different inbuilt tools that wasn't possible before.

s87
00:08:26.720 --> 00:08:33.039
For example, over here we're trying to get an agent to look up the internet to find the latest security reports for the React application.

s88
00:08:33.039 --> 00:08:37.360
We've given it access to the same index that Google has with type Google search.

s89
00:08:37.360 --> 00:08:43.360
We've given it the ability to be essentially retrieve information from web pages using the URL context tool that we ship with.

s90
00:08:43.360 --> 00:08:46.160
And we've given it a custom tool here called file incident.

s91
00:08:46.160 --> 00:08:53.920
The model can reason autonomously on its own, figure out what information it needs, and then actually generate a response for you all in a single API call.

s92
00:08:53.920 --> 00:09:04.240
This makes it very powerful because now you have models that not only can work with information at their training cut of date but beyond because now they have access to the entirety of the internet.

s93
00:09:04.240 --> 00:09:12.640
And all this boils down to the new steps data model that we have where instead of a world where we have a single message sent to a model and a response,

s94
00:09:12.640 --> 00:09:16.880
we're going to have very complex things like async tool calls models working together, right?

s95
00:09:16.880 --> 00:09:21.040
And we want to make sure we have a strong kind of data structure to support this.

s96
00:09:21.040 --> 00:09:30.399
And so we've introduced the steps data model which replaces the original legacy outputs array with a tight discriminator that makes it very clear at each step what the model output

s97
00:09:30.399 --> 00:09:36.480
what the dot signatures were what the functions called were and especially when it comes to the content is being generated.

s98
00:09:36.480 --> 00:09:45.760
Earlier you saw content.type where it was audio and video and these makes it easy to build these multimodal pipelines in a way that only the Gemini model families can.

s99
00:09:45.760 --> 00:09:51.279
The second thing that we've launched is anti-gravity as a remote agent as what we like to call manage agent.

s100
00:09:51.279 --> 00:09:56.560
So we've standardized around using the anti-gravity harness across all of our Google suite of products.

s101
00:09:56.560 --> 00:10:00.240
In AI studio, which you saw just now, websites are built with the anti-gravity agent.

s102
00:10:00.240 --> 00:10:03.920
In the anti-gravity application, you're also using the anti-gravity agent.

s103
00:10:03.920 --> 00:10:07.600
Now, this is a very capable harness that we've co-rained Gemini on.

s104
00:10:07.600 --> 00:10:12.800
And moving forward, I think you're going to see many gains if you use this harness out of the box.

s105
00:10:12.800 --> 00:10:16.399
If you wanted to build a coding agent in the past, you would first have to tune the harness.

s106
00:10:16.399 --> 00:10:24.320
Then you had to find a sandbox provider, manage the infrastructure, and then kind of figure out way how to preserve the context, especially in between runs.

s107
00:10:24.320 --> 00:10:29.519
But with the anti-gravity manage agent, you get a persistent sandbox out of the box.

s108
00:10:29.519 --> 00:10:37.120
With a single API call, you get a sandbox that you can keep hitting and treat as a personal claw without any sort of modifications.

s109
00:10:37.120 --> 00:10:41.600
You can see an example over here where we want to analyze what's in a GitHub repository.

s110
00:10:41.600 --> 00:10:51.839
We then pass it over in an API call over to the anti-gravity agent and it then boots up a remote sandbox and then starts investigating what the repository is about.

s111
00:10:51.839 --> 00:10:55.040
You can see it calls a list file to get what's in the workspace.

s112
00:10:55.040 --> 00:10:59.440
It then reads each individual file and then it generates a final report.

s113
00:10:59.440 --> 00:11:03.600
All of this is happening in a remote sandbox without any sort of intervention from our site.

s114
00:11:03.600 --> 00:11:08.959
The agent's able to research, reason and handle all of this autonomously.

s115
00:11:09.120 --> 00:11:19.440
We also handle multi-state persistent sandboxes like what we mentioned earlier where as long as you preserve the environment ID that we returned to you, you're able to route it right back to the original sandbox that you had.

s116
00:11:19.440 --> 00:11:21.279
So these are two primitives that we given you.

s117
00:11:21.279 --> 00:11:30.240
The interaction ID where you're able to preserve and work with context and ensure you're not busting your cache and the environment ID that enables you to route it to the exact same context.

s118
00:11:30.240 --> 00:11:32.160
And so this way you can use the same sandbox.

s119
00:11:32.160 --> 00:11:37.839
You don't need to manage any sort of persistence of state and all you need to do is just add a few lines of code.

s120
00:11:37.839 --> 00:11:41.760
The other way we've tried to make it easy is for you to be able to load in sources.

s121
00:11:41.760 --> 00:11:43.920
A lot of times you might have custom dependencies.

s122
00:11:43.920 --> 00:11:48.959
You might have EMD files, helper files that you want to throw on the on this specific sandbox.

s123
00:11:48.959 --> 00:11:51.519
So we support Google GCS buckets.

s124
00:11:51.519 --> 00:11:56.079
We support GitHub repositories and inline files.

s125
00:11:56.480 --> 00:12:01.040
And so what you see over here is a demo whereby it continues off the previous call that we did.

s126
00:12:01.040 --> 00:12:06.560
We're passing in the environment ID and the model has the exact same files right where we left it.

s127
00:12:06.560 --> 00:12:16.800
any sort of packages you install, any sort of files you throw and create, the model has access to it through each and every turn as long as you passes back the interaction ID and the environment ID.

s128
00:12:16.800 --> 00:12:21.120
This isn't just the sort of constraint to very simple task with the manage agent.

s129
00:12:21.120 --> 00:12:32.240
What you're seeing over here is a run where the model burnt over two million tokens trying to analyze a repository that we got for a hackathon where someone built from scratch an entire programming language to code reinforcement learning environments.

s130
00:12:32.240 --> 00:12:38.079
If you've ever used the Scratch programming language, it's a drag and drop language where you can kind of get simple characters to move.

s131
00:12:38.079 --> 00:12:49.040
But over here with a single API call, we're able to analyze an entire repository that has a custom DSL, a web page, a whole bunch of like schemas, and an RL training loop running inside HUD.

s132
00:12:49.040 --> 00:12:58.320
And so it really opens up this kind of opportunities for you to build these very custom and performant agents without you having to handle the underlying infrastructure or tune the harness.

s133
00:12:58.320 --> 00:13:11.120
The best part about using the anti-gravity agent is that fundamentally it's the same agent running in your anti-gravity IDE, which means that if you have certain workflows that are you've tuned locally, you have skills that you've made sure are great,

s134
00:13:11.120 --> 00:13:15.120
you can once you're happy with those skills, you just package it into a single folder.

s135
00:13:15.120 --> 00:13:26.560
We allow you to upload the same way using the sources into into agents folder and that gives your agent the exact same skills with the same agent harness and the same prompt running locally and in the cloud.

s136
00:13:26.560 --> 00:13:32.880
When you're ready or when you're comfortable with what your agent is doing locally, you can then push it up and everything works as intended.

s137
00:13:32.880 --> 00:13:38.320
The next question we often get from companies are, "Hey, I need my agent to be secure.

s138
00:13:38.320 --> 00:13:41.519
What am I going to do if my agent leaks a network credential?"

s139
00:13:41.519 --> 00:13:44.639
And to do so, we've implemented a man-in-the-middle proxy.

s140
00:13:44.639 --> 00:13:46.160
So, what does this mean?

s141
00:13:46.160 --> 00:13:50.800
A lot of times, you need to make API calls to different providers, say the Gemini API.

s142
00:13:50.800 --> 00:14:00.880
You might need to download private GitHub repositories, etc. We have a proxy that stands in the middle and so any call your model makes we will look at the header and we can replace items

s143
00:14:00.880 --> 00:14:01.600
at will.

s144
00:14:01.600 --> 00:14:10.160
What you see over here is a case where we'll transform every single outbound call to the GitHub API with an inject in the token dynamically.

s145
00:14:10.160 --> 00:14:18.160
And so even if the model is somehow prompt injected and leaks your GitHub API token, it's never actually going to see the token.

s146
00:14:18.160 --> 00:14:24.160
It's only going to execute code that will run against the GitHub API and your token is never exposed.

s147
00:14:24.320 --> 00:14:34.560
Once you're comfortable with how the agent is performing, you've set it up well, we give you two different ways for you to basically configure and set it up as a named agent, much like you've been using the anti-gravity preview agent.

s148
00:14:34.560 --> 00:14:39.360
The first one is to freeze it with a certain set of configured files and network sources.

s149
00:14:39.360 --> 00:14:42.000
That's what you're seeing on the left here.

s150
00:14:42.000 --> 00:14:50.160
I have a data analyst agent that I'm creating from the anti-gravity preview base agent with some base environment configurations here, just a GCS skill.

s151
00:14:50.160 --> 00:14:56.079
The other way that we allow you to do so is by chatting iteratively with an agent to set up an environment as you like.

s152
00:14:56.079 --> 00:15:06.320
For example, you might want certain dependencies, you might want certain packages, and you can freeze an environment as a specific agent and all your subsequent calls kind of go through that.

s153
00:15:06.320 --> 00:15:09.680
Once you're happy with that, it makes it very easy for you to scale up your workloads.

s154
00:15:09.680 --> 00:15:16.639
At this point, we support up to a thousand named agents, which means that you can have all these different configurations and you don't pay for any of the storage.

s155
00:15:16.639 --> 00:15:19.040
You don't pay for you don't pay for the sandbox.

s156
00:15:19.040 --> 00:15:20.800
you only pay for the model.

s157
00:15:20.800 --> 00:15:24.880
We've optimized our infrastructure to be very performant to be able to handle a workload like this.

s158
00:15:24.880 --> 00:15:28.639
And we're pretty excited to see what people are going to build.

s159
00:15:29.120 --> 00:15:33.279
Recently, we've also out open sourced a new Gemini API CLI.

s160
00:15:33.279 --> 00:15:41.040
This is a simple CLI that you can use to get started working with the Gemini models to test out the models locally using simple CLI commands.

s161
00:15:41.040 --> 00:15:47.600
The other benefit of this is that you're going to be able to iterate on your code on your coding agent or your anti-gravity agent locally.

s162
00:15:47.600 --> 00:15:54.959
And once you're ready, it packages it as a single folder and uploads it straight and creates an agent for you.

s163
00:15:55.600 --> 00:16:04.000
Lastly, if you're thinking of switching over to the interactions API, which is where all of our new models will be on, we've also created a new interactions API skill

s164
00:16:04.000 --> 00:16:10.399
that you can feed to your favorite coding agent and that will then basically allow you to do the migration without any issues.

s165
00:16:10.399 --> 00:16:24.160
I'm sure you guys have faced issues where you've asked Gemini to implement a specific kind of uh implementation and it keeps using Gemini 2.5 flash or Gemini 2.0. And so we what we do is we make sure we keep up to date the list of all the different models.

s166
00:16:24.160 --> 00:16:26.079
We keep up to date all the different methods.

s167
00:16:26.079 --> 00:16:37.040
We provide a good bunch of skills and like Philip mentioned yesterday, we regularly evaluate these and make sure make sure that they're actually doing what we want them to do, which is allowing you to use Gemini and build on Gemini well.

s168
00:16:37.040 --> 00:16:38.560
So that's the end of my talk.

s169
00:16:38.560 --> 00:16:41.920
Uh, thank you so much for coming by and staying around for the round of the conference.

s170
00:16:41.920 --> 00:16:43.680
And yeah, I hope you guys get started.

s171
00:16:43.680 --> 00:16:47.440
I'll be on the side if you guys have any questions about the new API.

s172
00:17:00.501 --> 00:17:02.501
[music]