Why AI Agents Should Have Their Own Sandbox — Philipp Schmid, Google DeepMind

AI Engineer · 20 min · 199 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12So, hi everyone.
00:13Uh, welcome to why agents should have their own sandbox.
00:17Uh, I'm Philip.
00:17I'm part of the Google DeepMind team, mostly working on AI studio and Gemini API.
00:22Uh, we have a small crew, so I try to keep the boring stuff short and like the exciting demo stuff long.
00:28And we can also ask some questions I guess at the end.
00:31Maybe you need to sit a little bit further at the front so I can understand you.
00:35Um so I want to start with a little bit of a background.
00:38I'm not sure if you have seen that slide but that's a very popular slide when you do like Google presentations um to show a little bit um what we have achieved in the last two year with like the Gemini models but also our open models with Gemma and the slide is already outdated since yesterday.
00:55So yesterday we shipped Nano Banana Light uh which is our new image generation model.
01:00Super fast uh sorry super fast uh super cheap.
01:05Um it's like 4 cent per image and then also we shipped Omni flash on the interactions API.
01:11So you can now programmatically use Omni to create videos to edit videos to yeah build gen media applications.
01:20And to give a little bit of a brief history, I assume everyone has heard about agent has heard about large language models.
01:25But to really quickly reflect where we are coming from.
01:29So I think around 2018 19 with the first bird models and GBT models, we were super excited because we could continue sentences, right?
01:37We were able to generate poems without uh with providing like a first few words and then the model coherently continue the text which was nice but not very useful or helpful for any one of us to
01:49build something.
01:50Then we we moved into the the instruction uh way.
01:54Before we had pre-trained models which could continue text and then with instruction following we basically trained models to answer questions more or less.
02:02So we told ask a question and the model correctly responded.
02:06Um and then with with chat GPT um everything changed a little bit.
02:10I think most of the people probably got started with that and we went into this user model turnbased conversation where we had a user input, a model input, a user input, a model output.
02:20Then afterwards we combine that with function calling because we wanted the model to take actions for us.
02:25And now we are in like this black box with goals, thinking and functions where we no longer just prompt the model.
02:31We basically give a goal to the model, tell them how to verify its goal or how to like deploy it and like hopefully have it running for multiple hours or even days and come back with the results we want to achieve.
02:44Um, so if you haven't used Gemini API or AI studio before, you can go to AI.studio or toi.dev.
02:51So very quick, very short URLs.
02:54H, get your own API key.
02:55So if you have your computer with you and you if you want to follow along a little bit later on like trying out a few things we have shipped recently
03:02you can do that.
03:03Um no setup required can use your Gmail account.
03:06Um can start start experimenting.
03:08If you want to use bigger models more usage enter your credit card and we improved that a lot.
03:14So you don't need to leave AI studio anymore to go into Google Cloud.
03:17You can do everything inside of AI Studio.
03:19You can set up your own budgets that nothing gets out of control.
03:23And I want to start about or talk about the interactions API which is our new uh API for Gemini models and Gemini agents.
03:31And on the right side sorry left side we have some code snippets.
03:36So if you never have heard about the interactions API you can think a little bit about that's our answers to like the responses API from OpenI.
03:44It is much more developer focused, much more simpler, much more web developer friendly.
03:49And you can already tell in like the two snippets here uh to call a model like Gemini or to call an agent here like the anti-gravity agent, it's like literally the same interface.
04:00And this works for all of the modalities Gemini offers.
04:03So you have a model or an agent, you have an input.
04:06The input can be a basic string or it can be multimodal data.
04:10And then you have tools or environment which uh I'm very excited and why which we are going to talk about later why every model or agent should have its own box.
04:20Uh new with the interactions API is you have serverside state management.
04:23So if you don't want to manage the state on the client you can hand it over to the server.
04:28We have long running operations.
04:30So async requests basically you start something.
04:33If we talk about omni video generation takes quite a long time and you might not want to keep your HTTP connection open.
04:40So you send a request get back an ID can set up streaming or polling to wait uh until the the generation is completed and then um it comes with a lot of simplifications
04:52for building agents and tool use.
04:53So you can combine built-in tools like Google search uh with custom function calling and to look at some code examples.
05:00So for serverside state it looks very similar if you have used the responses API we call it interactions because we just not get a response anymore.
05:08We interact with models and agents.
05:10You send an input and if you want to do a follow-up turn you provide a new input and basically just the ID from the previous turn and then on the server we
05:19um concatenate all of like the different inputs and make sure that the model really reh sees what has been done so far.
05:27And this goes for all modalities.
05:29So for audio generation you have a model.
05:32We have our Gemini TTS model.
05:34We have our input and since we want audio back not text you define a response modality with audio and some generation configuration which was you for example want to use the same pattern works for image generation.
05:47So our image generation model uh nano banana here our input some generation uh generation config and then you get back your image and for tool use uh also much better I would say than previously.
06:03You just define your tools with like Google search URL context or your own custom function and then you send your request and the model decides on it on its own.
06:13do I first need to like run Google search or call the function and all of the code you see here is basically enough to create a very simple small CVE agent
06:23to detect if there are like any CVE um for react in this case and then like it uses Google search it uses URL context to visit those websites and then function calling to basically give structure to it so you can like save it in your database or look it up for example
06:39um and something new and maybe different to other model providers is that we moved away from this user model term principle.
06:49So if you build an application with another LLM provider or with open models, you have always this uh user model user model turn concept.
07:00So you as a user provide an input and expect the model to return an output.
07:04This worked until like one or two years ago I would say where we just got back responses from those models.
07:10But with agents and with reasoning, all of that changed, right?
07:14Models now think for a very long time where they might expose some thoughts directly or might just expose a signature.
07:21Function calling is not very uh a user role, right?
07:26With normal or like with previous APIs, you would need to use the user turn to return some function output, which for me always felt a little bit weird because the output or the new input is coming from the environment and not from the user.
07:39So that's why we moved away from like those turns into steps or like to um like synchronous timeline of steps where every action or generation is its own type which makes it very easy to extend to new features and capabilities which might come in the future but also makes it still like very simple to use because you
07:59really define what it is done.
08:02So if you build a function calling agent, you return a function result and not like a a user role user with some content which might be a function result or not.
08:12And at Google IO we ship the anti-gravity agent uh which comes with its own sandbox.
08:18So the anti-gravity remote agent inside the Gemini API is powered by the same agent harness which is used with the anti-gravity product.
08:26So if you have used the IDE or the new anti-gravity 2.0 O or the CLI, you might have used that harness already.
08:33Very important here.
08:34When we talk about harness, we don't mean the same agent.
08:38The harness is basically the the back end of our agent, but it might be different between different products.
08:43So the anti-gravity IDE is a coding agent.
08:46So it might have a little bit of different tools or a little bit of of a different system instruction than the anti-gravity agent we have in the Gemini API, but it's still the same harness.
08:55Meaning if we improve Gemini to become better with that harness, we benefit inside anti-gravity, but we also benefit inside of the Gemini API.
09:05And with that new agent, we also introduce this uh environment field.
09:10And an environment here remote is basically for that API call, your agent gets a sandbox managed by us.
09:20Meaning if you send a request, our back end starts a cloud sandbox where the agent can run code, can create files, write files, um, install dependencies, install your own tools or maybe call your own APIs
09:35and it's all managed and stateless on our side, meaning that you just send the API request.
09:40You don't need to manage the infrastructure and those environments stay persistent.
09:45So if you have multi-turn conversation where the agent um you set user input I don't know like um research the latest tech news and create a file and then you want to do like a follow-up interaction
09:58you can like use the same environment from your previous turn with that new interaction but also this means that you can share environments between different agents.
10:07So if you have like three agents working inside the same environment, they can share context through the file system or you can like even use sub agents to basically pick up a work uh another agent has been done and then of course environments
10:22are very good but the model needs the right context and the right information or the right tools to work inside those environments.
10:29So you can provide sources uh sources can be um GCS buckets, GitHub repositories or inline files.
10:35It's very useful if you want to integrate skills.
10:38So the anti-gravity agent is skills or file system native I would say.
10:44So if you store skills inside the skills directory, it will automatically picked up and uh be pro uh included into the system instruction.
10:52But also if you have an agents MD file from your coding agent or from your existing agents, it will also be automatically appended to the system instruction.
11:00and then um you can create an agent because like sending a request is nice and having all of the configuration stateless is also very good but what if you want to share it or what if team one builds an agent team two wants to use
11:14um therefore we created the agents API where you can define the base agent so what's the underlying brain so to speak you can define instruction and then also the
11:24uh environment um and the environment can be a raw environment like with sources or you and uh use an existing environment.
11:32So you can think about I have some kind of a scaffolding to do where I need to install like the GitHub CLI uh add some skills and maybe do some like sanity checks.
11:42I can use the agent to create its own environment.
11:46And after the agent is done, I get back an environment ID.
11:49I can use that environment ID to create the new agent.
11:54And every time I call this new agent, it will start from that already pre-loaded, preconfigured environment.
12:02And so it's like very easy for you to use agents to build their own environments and then like to create agents out of them which you can share with your users or with your customers
12:12without writing any Terraform without thinking about Kubernates, microservices, firecracker or anything else.
12:20And to make it easier for you and for your coding agents, we created a CLI for the Gemini API.
12:26So it's it's not a coding agent.
12:28It's not Gemini CLI.
12:30It's really an a CLI that your agent can use to call the Gemini API.
12:36Meaning it can call Nano Banana or Omni if you want to generate some assets.
12:40But you can also use it to create those agents, iterate on those agents, and improve those agents.
12:45The QR code should bring you to like the GitHub repository and then also we have dedicated skills for working with the interactions API and the agents API um if you want to to play around with it later.
13:01And now let's let's do some some demos.
13:06So I'm in AI Studio inside the AI talk radio.
13:11If you have seen or attended the Google IO developer keynote, you might have seen uh this already.
13:18Um that's just a simple UI on top of managed agents.
13:22We manage agent.
13:23We are going to look at the agent in a little bit, but it's basically a talk show radio and that talk show u agent has access to all of the different Gemini models.
13:32So it can generate a thumbnail, it can generate um speech, it can generate music, it can do research and it's basically a one short radio generator and we give it a try.
13:45And I want a talk show radio on whether pineapples belong on a pizza or not.
13:57And it's free to use.
13:58So if you go to the applet later uh you can like have free daily usage to create create your own um AI talk show radio and what happens is like first step we start our environment and then the model starts thinking on what it needs to do and then the first step is like it installs our Python
14:16dependencies and it reads our skills and then it continues to work.
14:20Since this will take a little bit, I already ran a talk show radio on whether we should put in cereal first or milk first.
14:29And the very interesting part about a talk show radio, it's not like a podcast you listen to.
14:34It's really you have multiple um people like reasoning and like sharing their opinions on it and we give it a quick listen.
14:43Is there a correct physical order to preparing your breakfast bowl or are we flirting with culinary chaos?
14:50[music]
14:51Welcome to AI Talk Radio.
14:54I'm Paul broadcasting from our London studio.
14:57Today we are exploring the great breakfast debate.
15:01Cereal first versus milk first.
15:04We will objectively break down the pros and cons of both methods.
15:08Let's go to our lines.
15:09We have David calling from Boston.
15:12David, you're on the air.
15:14Yeah, thanks Paul.
15:16Look, it's got to be cereal first.
15:18It's basic logic, right?
15:20Cereal is the independent variable.
15:23You pour it, you see the volume, then you add milk to compliment it.
15:27Plus, like if you pour dry cereal into a bowl of milk, it splashes everywhere.
15:32It's a total mess.
15:34And the mashed potato analogy, you wouldn't put gravy in a bowl and drop a lump of potatoes on top, right?
15:41A striking analogy, David.
15:43Cleanliness and portion control are certainly strong arguments.
15:47Let's hear the counterpoint.
15:48We have Sarah on the line from Sydney.
15:51Sarah, welcome.
15:53Hi, Paul.
15:54Okay, look, David has it all wrong.
15:56Milk first is it's a total game changer for texture.
16:00Okay, cool.
16:01So, if you want to play around with it yourself, you can go to AI.studio.
16:07And on AI.
16:08studio you have that build section on the left side and there's a gallery and gallery is basically pre-built applets use cases for Gemini and if you scroll down to it uh you have new build experience with and there's the talk show radio if you click on it you will find the applet
16:25you can uh remix it meaning you can like adjust it so if you don't like talk show radios and you would rather have I don't know some real podcast
16:34uh application you can like very easily together with Gemini adjust it to their own needs.
16:39And since applets are all open uh in terms of code, we can also take a quick look on like how our agent really is.
16:47And our agent is just files.
16:50So we have an agents MD file inside our environment.
16:53That file will be loaded on environment startup which is then included into the system instruction.
16:59Very basic agent file.
17:01It has all of like the instructions for our agent to do.
17:04So it should do some research, write some scripts for the different people, generate music, generate a speech, mix everything together.
17:12And to do all of those works, we have skills for it.
17:15So if we look into like the TDS generation, we have a skill on how to run it.
17:19We have a Python file.
17:21That Python file itself allows the Gemini agent to call other Gemini models.
17:26So it uses the interactions API to generate some speech which we can then use.
17:31And that that's all it takes for our agent.
17:34And if you want to take that agent from the UI to like a back end, it's really you just need to copy that file and create the agent and then use it.
17:43And if you want to play around with the uh agent itself uh inside the playground, we have uh inside the model pickers a new agent section.
17:53So if you want to test deep research, go test deep research.
17:56But there's also the anti-gravity agent.
17:58And on the anti-gravity agent, we have six different uh agent templates uh which you can like very easily test, select and and see how it works.
18:06We go into like the the standard one.
18:08And what I really like to do is like explore the environment like let's see what the agent finds out about where it is running.
18:15We send our prompt and now it starts the environment should take for the first request like 7 to 10 seconds and then we see the agent like doing its work.
18:24So we have some reasoning, we have some bash commands.
18:27So it runs so it checks what uh version west version I'm running on what CPU memory I have and then we get back a response from our agent that is runs on Ubuntu
18:38um has 8 vCPU 16 GB of memory python installed pip installed and then you can like really continue let's see um do you have do we have node installed
18:53um so you can like very easily follow up um and it just continues based on like my prompt and then also you have the option on the right side to
19:03download the environment.
19:04So if your agent has created some files you want to extract out of the environment, share it, can very easily download it or there's an API for it as well.
19:12And then um uh what else?
19:16Uh to quickly show you again how it works, I can make that a little bit bigger.
19:21So when you when we send a request um we like start the sandbox and it loads all of our files and then we have like this agent loop completely on our server.
19:34So when you build agents normally a lot of times you need to manage the function calling on your client right you need to passse the function call run it some somewhere and then send back the function result and with the managed agents it's really
19:47just a single API call and everything runs inside that remote sandbox you can also connect remote MCP servers or run uh local MCP servers inside of that environment and
20:00um yeah really get started so if you are interested about it uh we have great documentation inside the Gemini API under agents with a quick start which walks you through how to send your first request,
20:11build your first agent and get started and then of course uh install the coding skills to have claude codeex or whatever model you use to to help you get started.
20:21Uh thank you for for coming.
20:22If you have some questions I'm like here