An Interaction Is All You Need — Ivan Leo, Google DeepMind

AI Engineer · 17 min · 172 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hey guys, my name is Ivan.
00:13I'm on the developer experience team at Google Demine.
00:16And today we're covering the brand new interactions API and our new offering manage agents.
00:20And so to take a quote from a very famous paper, an interaction is all you need.
00:25Before we move on to what the interactions API is and what the new manage agents offering is let's take a trip down memory lane and let's kind of trace how models have changed and how we've worked with models.
00:35So when we first started working with models a lot of times you would have a single interaction you would send a message tell me a joke and then you might get a response for example
00:44why don't scientists trust atoms because they make up everything.
00:48After a while as models got more and more capable and they understood more about the world this became a bit more complicated.
00:54We wanted to take these models, put them in different applications, but to do so, we needed them to be reliable.
00:59And so to do so, we invented function calling, where models could create individual JSON objects with predictable structures.
01:06Think like how when you want to register for a website, your web page sends a JSON payload to the back end with a username and a password.
01:14And that enables the backend server to say, okay, we have a new user and now we can actually create a new account.
01:20that made it possible for us to actually use models in different parts of this application whether it's extracting user information, parsing data and more.
01:28But right now we're seeing this incredible cups where models are getting more capable.
01:32We're seeing models being able to reason go for very extended periods of time before they come back to people.
01:38And so you're moving from a place where we went from completions instruction and functions where we might ask, hey, what's the weather in New York City?
01:45and the model has a call weather tool that it can get that data, ingest it, and generate a response to now a case where models have multiple tools under their belt.
01:54They're able to call tools, think and reflect on what the response meant, interact with the environment, and ultimately generate a response.
02:02And so this just brings us back to what is an agent.
02:06Fundamentally, if you look at what an agent is, it's really a model that powers a huge chunk of it.
02:11A lot of people have reduced this to just a language model running in a loop.
02:15And for most cases, this is true.
02:18If you have a really capable model, you'll find that as we've got models that become more and more capable, a lot of the scaffolding has fallen away.
02:25A lot of models, for example, the latest Opus models or fable models are now just immediately just using a bash tool instead of individually specialized tools like a edit file or a read file tool like we used to do.
02:36And so really what we have is a language model that's used to understand the context, the user's intent that's given a bunch of different functions, all different resources that it then interacts with its environment in.
02:48It has some short-term memory and some long-term memory to keep track of what it's encountered and then an environment to execute functions in.
02:55If you're building a coding agent, but you don't give it an environment where it can actually execute code, for example, it's not going to be able to do anything.
03:02And so if you'd like to try any of our models, the easiest way to do so is actually with AI Studio.
03:08And so I actually like to point out that my speaker notes today were actually coded on AI Studio.
03:12I vived it out 2 minutes before this talk.
03:15Uh you'll see the um you can see the conversation history over here.
03:19Um I literally said I need a mobile responsive website that I can pass in speaker notes because I found out I wasn't going to be able to use presenter view.
03:27And so what I did, I just gave it some brief instructions and before you know it, you have a simple application that you can use without any hassle deployed on a website that's accessible to all.
03:37So we have a really generous uh free tier for most of my presentations.
03:41I've just been using the free tier.
03:43I haven't been using my Google unlimited tokens thankfully.
03:46And I think it's the best way for you to try our models.
03:48Whether it's our new speechtospech translation models where you take in some you speak in let's say English and it can translate to any language or you know you want to try our latest models like the Gemini 3.5 flash etc. It makes it very easy for you to do so with simplified UI toggles to just be able to
04:04tune all of the different parameters.
04:06So let's talk about the interactions API.
04:08Now one of the things that we've seen especially as we've moved from models to a dynamic where now we have models and agents is that the kind of workloads that we want to support have changed significantly
04:21right through the Gemini API you have nano banana you have you know Gemini 3.5 flash but you also have very capable agents like the deep research agent that's going to run for anywhere from three minutes and more
04:32you interact with it you have some sort of research plan it's going to go out and do its own thing and it comes like a very deep and detailed
04:39research reports.
04:41Now, it's pretty difficult then to think about consuming models and agents through a whole bunch of these different endpoints.
04:48Additionally, for a lot of the older kind of API endpoints, a lot of the data was in in very deeply nested objects.
04:55And so you can imagine the kind of difficulties that we had ourselves when we were building in first-party integrations.
05:00And we wanted these use cases where you might be able to generate a nanobanana image, do some research, generate a vo image from there or spawn out all these different agentic
05:10kind of like agents that go out and do their own thing.
05:12And so the interactions API is our sort of solution to this as we look to build an API that can support all of these different use cases and ones that we can build on ourselves.
05:22you look to bring Gemini across the entire surface of the Google ecosystem.
05:28So the best place to get started is just with serverside state.
05:32One of the things that you'll realize especially as you use the new Gemini series of models is that we essentially have this thing called thought signatures where if you interact with the model, if it calls a function, if it gives a response,
05:42it's going to pass back a very opaque series of these numbers.
05:46For a lot of people, manually managing this was very difficult and sometimes when we talked to different startups, we found that they would lose their cache just with a single whites space that they mistakenly added in.
05:56So now with the responses API, you can see on the first turn, hi, my name is Phil.
06:00And in the response, we give you a interaction ID.
06:03And as long as you give us back that same interaction ID under the parameter previous interaction ID that you'll see in the second turn, all that context is preserved.
06:13And this is really important with the new Gemini series of models because if you don't give us back those thought signatures, you're going to see a decrease in performance.
06:21But what else does this unlock?
06:23Here you see a demo built by Vampsy from our team that did an incredible demo where using a single photo, you could generate with Nanovana all these different variations of yourself in different countries and different locations.
06:35Each of these individual photos are a single interaction, but you can think of it from an initial API call.
06:39It spawns all these different interaction calls.
06:42Using those interactions, the same interaction ID, we then generate a video using the interaction ID and the new Omni model that we just launched.
06:50Something like this would not have been possible without the interactions API built by Philip over there.
06:55So, actually, it's a really incredible piece of work, I will say.
07:01So, to give you an example, an idea of what it would take to build something similar, this is all it would take.
07:06First, you pass in Gemini 3.1 flashlight image, generate an image of me in Venice.
07:11You take the interaction ID, you chunk it back into the client.create, and this time you specify that you want the Omni flash model instead that we launched on Wednesday.
07:21And that allows you to now use the same Omni model with the same information, the same kind of images, and the same context throughout all these calls.
07:29It's making it possible to build more complex, rich, and just these really complex agentic applications.
07:36The same thing occurs with audio generation.
07:38If you look at the bottom, you see that now we have output.type.
07:42In the past, you would have to take this very nested object like completions audio.
07:48And it would be very tricky for you to nest it out.
07:50I can't even remember it off the top of my head.
07:52But now, every single bit of the outputs that come out are clearly demarcated by a strong type.
07:57If it's an audio, as you see over here, all you got to do is just parse it and handle it accordingly.
08:02Right?
08:03But what happens if now I want to generate an image?
08:05It's the same method over there, client.interaction.create.
08:09All I got to do is just modify the response modality and the generation config.
08:14And now I have an output.type of image.
08:16And this makes it very easy for me to start building and experimenting with Gemini.
08:21The same thing comes with tool use.
08:23You can mix and match different inbuilt tools that wasn't possible before.
08:26For example, over here we're trying to get an agent to look up the internet to find the latest security reports for the React application.
08:33We've given it access to the same index that Google has with type Google search.
08:37We've given it the ability to be essentially retrieve information from web pages using the URL context tool that we ship with.
08:43And we've given it a custom tool here called file incident.
08:46The model can reason autonomously on its own, figure out what information it needs, and then actually generate a response for you all in a single API call.
08:53This makes it very powerful because now you have models that not only can work with information at their training cut of date but beyond because now they have access to the entirety of the internet.
09:04And all this boils down to the new steps data model that we have where instead of a world where we have a single message sent to a model and a response,
09:12we're going to have very complex things like async tool calls models working together, right?
09:16And we want to make sure we have a strong kind of data structure to support this.
09:21And so we've introduced the steps data model which replaces the original legacy outputs array with a tight discriminator that makes it very clear at each step what the model output
09:30what the dot signatures were what the functions called were and especially when it comes to the content is being generated.
09:36Earlier you saw content.type where it was audio and video and these makes it easy to build these multimodal pipelines in a way that only the Gemini model families can.
09:45The second thing that we've launched is anti-gravity as a remote agent as what we like to call manage agent.
09:51So we've standardized around using the anti-gravity harness across all of our Google suite of products.
09:56In AI studio, which you saw just now, websites are built with the anti-gravity agent.
10:00In the anti-gravity application, you're also using the anti-gravity agent.
10:03Now, this is a very capable harness that we've co-rained Gemini on.
10:07And moving forward, I think you're going to see many gains if you use this harness out of the box.
10:12If you wanted to build a coding agent in the past, you would first have to tune the harness.
10:16Then you had to find a sandbox provider, manage the infrastructure, and then kind of figure out way how to preserve the context, especially in between runs.
10:24But with the anti-gravity manage agent, you get a persistent sandbox out of the box.
10:29With a single API call, you get a sandbox that you can keep hitting and treat as a personal claw without any sort of modifications.
10:37You can see an example over here where we want to analyze what's in a GitHub repository.
10:41We then pass it over in an API call over to the anti-gravity agent and it then boots up a remote sandbox and then starts investigating what the repository is about.
10:51You can see it calls a list file to get what's in the workspace.
10:55It then reads each individual file and then it generates a final report.
10:59All of this is happening in a remote sandbox without any sort of intervention from our site.
11:03The agent's able to research, reason and handle all of this autonomously.
11:09We also handle multi-state persistent sandboxes like what we mentioned earlier where as long as you preserve the environment ID that we returned to you, you're able to route it right back to the original sandbox that you had.
11:19So these are two primitives that we given you.
11:21The interaction ID where you're able to preserve and work with context and ensure you're not busting your cache and the environment ID that enables you to route it to the exact same context.
11:30And so this way you can use the same sandbox.
11:32You don't need to manage any sort of persistence of state and all you need to do is just add a few lines of code.
11:37The other way we've tried to make it easy is for you to be able to load in sources.
11:41A lot of times you might have custom dependencies.
11:43You might have EMD files, helper files that you want to throw on the on this specific sandbox.
11:48So we support Google GCS buckets.
11:51We support GitHub repositories and inline files.
11:56And so what you see over here is a demo whereby it continues off the previous call that we did.
12:01We're passing in the environment ID and the model has the exact same files right where we left it.
12:06any sort of packages you install, any sort of files you throw and create, the model has access to it through each and every turn as long as you passes back the interaction ID and the environment ID.
12:16This isn't just the sort of constraint to very simple task with the manage agent.
12:21What you're seeing over here is a run where the model burnt over two million tokens trying to analyze a repository that we got for a hackathon where someone built from scratch an entire programming language to code reinforcement learning environments.
12:32If you've ever used the Scratch programming language, it's a drag and drop language where you can kind of get simple characters to move.
12:38But over here with a single API call, we're able to analyze an entire repository that has a custom DSL, a web page, a whole bunch of like schemas, and an RL training loop running inside HUD.
12:49And so it really opens up this kind of opportunities for you to build these very custom and performant agents without you having to handle the underlying infrastructure or tune the harness.
12:58The best part about using the anti-gravity agent is that fundamentally it's the same agent running in your anti-gravity IDE, which means that if you have certain workflows that are you've tuned locally, you have skills that you've made sure are great,
13:11you can once you're happy with those skills, you just package it into a single folder.
13:15We allow you to upload the same way using the sources into into agents folder and that gives your agent the exact same skills with the same agent harness and the same prompt running locally and in the cloud.
13:26When you're ready or when you're comfortable with what your agent is doing locally, you can then push it up and everything works as intended.
13:32The next question we often get from companies are, "Hey, I need my agent to be secure.
13:38What am I going to do if my agent leaks a network credential?"
13:41And to do so, we've implemented a man-in-the-middle proxy.
13:44So, what does this mean?
13:46A lot of times, you need to make API calls to different providers, say the Gemini API.
13:50You might need to download private GitHub repositories, etc. We have a proxy that stands in the middle and so any call your model makes we will look at the header and we can replace items
14:00at will.
14:01What you see over here is a case where we'll transform every single outbound call to the GitHub API with an inject in the token dynamically.
14:10And so even if the model is somehow prompt injected and leaks your GitHub API token, it's never actually going to see the token.
14:18It's only going to execute code that will run against the GitHub API and your token is never exposed.
14:24Once you're comfortable with how the agent is performing, you've set it up well, we give you two different ways for you to basically configure and set it up as a named agent, much like you've been using the anti-gravity preview agent.
14:34The first one is to freeze it with a certain set of configured files and network sources.
14:39That's what you're seeing on the left here.
14:42I have a data analyst agent that I'm creating from the anti-gravity preview base agent with some base environment configurations here, just a GCS skill.
14:50The other way that we allow you to do so is by chatting iteratively with an agent to set up an environment as you like.
14:56For example, you might want certain dependencies, you might want certain packages, and you can freeze an environment as a specific agent and all your subsequent calls kind of go through that.
15:06Once you're happy with that, it makes it very easy for you to scale up your workloads.
15:09At this point, we support up to a thousand named agents, which means that you can have all these different configurations and you don't pay for any of the storage.
15:16You don't pay for you don't pay for the sandbox.
15:19you only pay for the model.
15:20We've optimized our infrastructure to be very performant to be able to handle a workload like this.
15:24And we're pretty excited to see what people are going to build.
15:29Recently, we've also out open sourced a new Gemini API CLI.
15:33This is a simple CLI that you can use to get started working with the Gemini models to test out the models locally using simple CLI commands.
15:41The other benefit of this is that you're going to be able to iterate on your code on your coding agent or your anti-gravity agent locally.
15:47And once you're ready, it packages it as a single folder and uploads it straight and creates an agent for you.
15:55Lastly, if you're thinking of switching over to the interactions API, which is where all of our new models will be on, we've also created a new interactions API skill
16:04that you can feed to your favorite coding agent and that will then basically allow you to do the migration without any issues.
16:10I'm sure you guys have faced issues where you've asked Gemini to implement a specific kind of uh implementation and it keeps using Gemini 2.5 flash or Gemini 2.0. And so we what we do is we make sure we keep up to date the list of all the different models.
16:24We keep up to date all the different methods.
16:26We provide a good bunch of skills and like Philip mentioned yesterday, we regularly evaluate these and make sure make sure that they're actually doing what we want them to do, which is allowing you to use Gemini and build on Gemini well.
16:37So that's the end of my talk.
16:38Uh, thank you so much for coming by and staying around for the round of the conference.
16:41And yeah, I hope you guys get started.
16:43I'll be on the side if you guys have any questions about the new API.
17:00[music]