Build Your Own Team of Managed Agents - Jack Wotherspoon, Google DeepMind

AI Engineer · 19 min · 213 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everybody.
00:12Thank you so much for coming.
00:14My name is Jack Witherspoon and I am a DevEx engineer at Google DeepMind and today we're going to talk about managed or cloud agents.
00:23So raise your hands here, who all has heard of either cloud or managed agents or has used them?
00:30Okay.
00:31I'm starting off on the right foot.
00:34So you might have heard of them if you for those who haven't, cloud or managed agents are essentially a way of instead of running a harness and running coding agent on your own machine,
00:45you can actually just run them in the cloud on a sandbox.
00:50So let's first start by why the heck would you even want to do that?
00:54Well, sandboxes are really powerful for a number of reasons.
00:58Number one, they're not running on your own machine.
01:00So you can, you know, spawn a cloud agent.
01:04It can go and write code.
01:05It can reason.
01:06It can do research, search the web.
01:09It can go ahead and write tests, run tests, verify and actually come back with verified kind of answers.
01:16And that's a really strong thing.
01:18If you've ever like tried to run verification either on your own laptop, you know, sometimes it steals the browser.
01:24It starts to run things for you.
01:25You start getting pop-ups and things like that.
01:27So running this all in the cloud takes away that pain.
01:32And the other reason is that your laptop doesn't scale.
01:36You might start to have specific setups like agent skills or you maybe have customized your personal coding agent on your machine.
01:44But then how do you go and share that with a colleague or how do you integrate that into your app, right?
01:50You don't want your app that you're deploying to customers to be running with your own kind of subscription or things like that, right?
01:57So, we want to be able to scale this up and doing so.
02:01And managed agents or cloud agents are a great great way to do that.
02:05So close, the sign almost stayed up.
02:08Don't worry about it.
02:09[laughter]
02:12So, when it comes to kind of managed or cloud agents, you know, to do this yourself is quite a tedious process.
02:20You know, you have to actually spin up a container or a VM.
02:23You have to load things onto that, manage it.
02:27And then you actually have to do harness engineering, you know, figure out kind of build your own harness within the sandbox.
02:33And then let's not even start to talk about networking or setting up firewall rules or all the security stuff.
02:39So, that's where kind of managed agents come in and whether it's Gemini managed agents, cloud managed agents, we'll be talking about Gemini managed agents today, which is our offering
02:48in the Gemini API at DeepMind.
02:51And really now it boils down to just being a single API call.
02:57So, that is really the beauty of managed agents.
03:00With a single API call, you're getting all the things we talked about.
03:04So, you're getting the sandbox, you're getting the anti-gravity harness built out of the box that is in the sandbox, and it can go ahead and do a lot of powerful things.
03:13You don't have to worry about managing any of the infrastructure or the scale as you need in your application more agents, you can just do more API calls.
03:26Actually, we'll go back for a sec.
03:28So, you can see here, this is our interactions API.
03:31It's our kind of new agent first version of our SDKs and client libraries.
03:37So, previously we had kind of models, you would, you know, do an input, get an output, but that didn't really scale to agents in the new world we live in.
03:47So, we launched a new kind of SDK uh, covering interactions.
03:51So, interactions are much more kind of mirroring what agents are doing.
03:55So, it can be multiple steps, you know, when a when you do an agent call, it can go ahead and write files, call those files, call different MCP tools.
04:04Um, so really this new interactions API that we launched, uh, and went GA earlier this year, is now our kind of home for managed agents.
04:13So, you just call the agent instead of having a model, you have an agent.
04:16So, we have our base anti-gravity agent.
04:19And then you set environment equals remote.
04:21So, that's that key thing that's now telling it, "Okay, we want a managed agent that runs in the cloud."
04:27And then you give it kind of regular input as you're used to.
04:32So, here's kind of how the infrastructure actually looks under the hood.
04:36So, when you call a managed agent, you kind of have that input that we saw.
04:39And then it calls the Gemini API, which has a bunch of, you know, built-in server-side tools like, um, searching the web through Google Search or file system tools to read and write files.
04:50Um, and you can also add your own custom functions or MCP tools as well.
04:56And then we have the sandbox.
04:58So, that's actually running on kind of Google Cloud infrastructure.
05:01Um, so when you call the Gemini API to get a managed agent, it goes ahead and actually sets up and spins up that sandbox for you.
05:08And you can go ahead and actually upload files to the sandbox.
05:11So, if you have agent skills or agents.md, or you want to load a GitHub repository on startup, uh, we allow you to do all of that cool stuff.
05:22So, here are kind of the building blocks that make up a managed agent.
05:26These are kind of the different core pillars.
05:29Um, so you've got the harness itself, and that's the anti-gravity harness.
05:32It's the exact same harness that's running, uh, in our anti-gravity suite of products over at Google.
05:39So, that's what's kind of figuring out the tool calls.
05:41It has the built-in tools.
05:43Um, and then we have the environment.
05:45We'll talk a bit more about how the environment is kind of its own entity, um, which allows you to do really cool kind of multi-agent workflows by having different agents pointed at the same sandbox so they can write files.
05:57A different agent can pick that same environment up later and actually build upon those files.
06:02And then we've got skills we talked about so you can upload your skills to the sandbox just as you would have potentially like playbooks or different ways of doing things at your company or on your personal agent, you can upload those to the managed agent as well.
06:15And then we have credentials.
06:17Credentials is a new piece of managed agent which essentially lets us inject either kind of tokens or your MCP OAuth into the managed agent sandbox so you can access your different services.
06:29And then we also have triggers which allow you to actually schedule a managed agent to run.
06:34So let's say at 8:30 8:30 a.m. every morning before the market opens you want to go and you know do some kind of financial analysis before the market opens up.
06:43You could set a trigger.
06:45The managed agent would essentially wake up, run its task or run its input, output a result.
06:50You can like do webhooks so that you could send a Slack message.
06:54Maybe it queries your team's Jira board every morning and sends you know here's what everyone's working on and things like that.
07:02So we talked a little bit about the harness.
07:04It's the anti-gravity harness that runs under the hood and you can change the different models that it runs with.
07:10Yesterday we just launched Gemini 4 Argon which is our new frontier model.
07:14So I definitely recommend testing it out.
07:17It's a great model.
07:19But we'll show it later on in this in demo.
07:23And you can set the different model in the config for the managed agent in the agent config as you see there.
07:28We have that kind of one-line parameter to set the model.
07:31So that's how you can kind of tune the managed agent based on the workflow.
07:35If you're wanting to do something that needs to be fast and efficient, you can use Gemini Flash.
07:39If you want to do do that's deep reasoning or potentially doing a lot of analysis, then you can use a different model like Gemini 4 Argon.
07:49Like we said, Gemini 4 Argon was released yesterday.
07:52Um it's a really great model.
07:53It's especially good at financial tasks, which is great for hopefully for the audience here.
07:58Um and really good also at doing data analysis across a wide range of documents.
08:02Um and as always, the Gemini models are also super super good at multimodal.
08:07Another really cool, interesting um feature that we have with the Gemini 4 model is long decode.
08:14So, normally models have an output token max of around 64K tokens.
08:19Um but with Gemini 4, we actually have up this so that now the 1 million kind of context window or token limit is now shared between input, thinking, and output.
08:30So, output, as you need to kind of scale up if you need to do extreme kind of math reasoning or financial modeling where the output might be pages and pages and pages long,
08:40the model can actually go and do that.
08:41And this is showing to actually advance some of the more mathematical and financial benchmarks out there.
08:50So, we talked a bit about the interactions API and environments, and this is kind of one of the key pillars.
08:55These are kind of two core primitives in managed agents that allow you to get a lot of flexibility and be able to customize kind of the agent configurations that you do.
09:05So, these are kind of two different um stateful things on our back end in our server side.
09:11So, interactions is what manages your conversation.
09:15So, when you put an input and get an output and kind of a different turn with the model, um those are saved as kind of what we call interactions.
09:22So, you can take one interaction, pass that ID to the next interaction, and that kind of, you know, it's the second turn.
09:29Um and then we also have environments.
09:31So, environments are what's actually kind of in the file system on the sandbox.
09:37So, when the agent goes and writes files or creates images or does different file tasks on the actual sandbox in the cloud, that's kind of representative of the environment.
09:50And you can go ahead and use our APIs to download files from that environment or upload them.
09:56But having these two things kind of split out allows you to do some really cool things where you can actually see here that we are creating one agent interaction
10:06with our first code block.
10:08And it's going to go ahead and do some work and save a file.
10:11And then we can pass that environment ID to our next agent.
10:14So, it actually picks up.
10:16It has a very fresh kind of context window and conversation, but it's reusing the environment.
10:21So, it can go and, you know, say save the Q3 filings in our workspace.
10:25Maybe we're going to go, you know, fetch the fetch the annual reports, the filings.
10:30And then we're going to go and do some analysis, save that as a file.
10:33And then we can have a new agent come in, reuse that same exact environment, but this time have a fresh context window.
10:42Kind of a multi-agent workflow, which is really cool.
10:46So, there are a bunch of built-in tools already in the sandbox, like code execution, reading and writing files, as well as Google Search.
10:53But we also support, you know, remote SMTP servers, so you can bring your own servers.
10:58You can hook up to your favorite services.
11:00And really that takes the agent a step further and be able to actually do kind of daily tasks or things that you can build into your apps.
11:10And we talked a little bit about it, but you can also kind of mount files to the file system.
11:14So, you can upload your different agents.md files, your skills, and kind of run through the playbooks that you have your agents doing on your personal machine, but doing it in an environment where you can kind of reproduce
11:26and have that work done for you in the cloud.
11:29So, you don't have to worry about, you know, overriding any files or messing up things on your machine.
11:34Everything runs in the cloud, um and can just run as a process and get you back that output.
11:40So, here's a Here's a code sample where we're actually going to upload an agents.md example into our um sandbox, and then we're also connecting a GitHub repository.
11:50So, now you can actually have the agent look at your code, check for bugs and things like that.
11:57And then you can also save the agents.
11:58We also have an agents API, so you can do agents.create.
12:01So, once you've kind of refined your workflow and found an agent that works for you, you can save it so you can easily call it, you know, in multiple different times.
12:09You can iterate on it and then save it for later.
12:14And then you can just easily invoke your personal agent by calling it with its uh agent ID.
12:21So, another big part of the security aspect of managed agents is that secrets never actually enter the sandbox.
12:28So, our managed agents have a proxy that sits uh it's a kind of a layer between the sandbox and our API.
12:35And it actually what it does is it puts a placeholder kind of dummy token into the sandbox so that the agent can never kind of go rogue and say, "Oh, I see that I have this GitHub token laying around.
12:46Let me just see what else is on there."
12:48Um so, it can actually never do that because it never actually has your GitHub token.
12:53So, what it does is when it goes and calls the GitHub API, our proxy will then inject that token.
12:58It will match it to say, you know, it's API uh GitHub API.
13:02Um we're going to go ahead and inject this secret.
13:06Um so, that way the agent can never do anything that it shouldn't be able to.
13:12We talked quickly about triggers, but triggers are also really powerful because you can have them set to kind of a daily briefing every morning like we said, scan your emails,
13:22maybe suggest ones to reply to, or if you want to check on the stock market before it opens, um you can do that.
13:28And this is a way to run one of your agents on a set schedule.
13:37And we talked a little bit about multi-agent workflows, but managed agents can actually work really well um in multi-agent workflows.
13:44So, you can actually set it up so that one managed agent knows how to, because it's just an API call, actually call other managed agents um and do work and fan out work.
13:55And then you can have very kind of niche and specific targeted agents that have one core purpose.
14:01They do that purpose really well.
14:03Um you know, maybe it's drafting notes or doing research.
14:07Um writes it to a file, gets that information saved, and the next agent can then take over from that different input.
14:13And this kind of breaks it up so that you can have things really controllable and really make sure that you're doing specific tasks well instead of just giving a super super broad,
14:23you know, task to an agent and having it have no way of kind of measuring what it's actually doing.
14:32So, now we'll just quickly put this to the test.
14:35Um so, we can actually run the agents here and see what this actually looks like when you run one.
14:40So, we have an SEC kind of an SEC filing researcher that just researches companies.
14:44We have it set to Nvidia here.
14:47And we can go ahead and make call the API.
14:49So, like again, this is just an API call.
14:51It's really easy to build into your own either internal workflows or to actually publish in your services or apps to external folks.
15:00So, we can go see here.
15:03There are step traces.
15:05So, each thing the agent does um is saved as a step trace.
15:08So, you can see we're having function result for the right to file, code execution, and you can kind of trace the what the agent is doing and optimize as you go to kind of make sure that the workflow is straightforward.
15:21And then were we using the same environment ID that we talked about?
15:24So, because we're using the same environment ID in this scenario, we're actually having our agents pick up.
15:28So, we can have We had this SEC filing researcher that went and did research on Nvidia, gave us a bunch of metrics as the output.
15:37And then, we can now kick off this kind of quant agent, which will go take a look at the metrics because it's in the file system.
15:44It's reusing the same environment, the same sandbox.
15:46And now, it can pick up its task and its job from that um metrics.json, take it, run a bunch of Python code files to see and actually look at it make evaluation
15:57for Nvidia.
15:58And if we go here and look at the MD file, we can see the managed agent went and actually did research based on the metrics it got, and it's now kind of doing a forecast
16:08for the share price in the stock.
16:12And then, we can kick off a third agent just to kind of take that research and knowledge dump and turn it into a report.
16:19And if we had connected different MCP servers, like we could connect Google Docs, have it actually output and write to a Google Doc, or send a message in Slack so that you can really kind of create these customized workflows from managed agents.
16:34And then, you can also build managed agents super easily with the SDK into your own application.
16:39So, this is just a demo here where we have a managed agent that is actually um connected to our service.
16:45Um it's connected to our Google Drive where we dumped in a bunch of receipts of our annual kind of um company data, and it's going to go ahead and actually create this visualization
16:56um by from from the the data.
16:58And the managed agent is the one that's actually doing all the research.
17:01It has, you know, kind of a skill that will go out and tells it kind of how to do this interactive report.
17:07Um and then, it's just kind of reasoning through it and building this out for us.
17:17So, it's really customizable.
17:18You can really kind of anything you can do on your own personal coding agent, you can kind of turn it easily into a managed agent that you can either run as a company daily or set up in your own apps so that we have other ones on our on our Gemini API on AI Studio where you can
17:33kind of create games on the fly.
17:35You can research documents and do all kinds of things.
17:44So, the easiest way to get started today is to go over to AI Studio and get started for free over there.
17:50And we also have our documentation for managed agents, but I'll quickly just do another demo here since we have a few minutes.
17:57This is one of our kind of managed agents examples that is on AI Studio in our gallery.
18:03And this one's kind of just trying to showcase how you can actually build these managed agents into your own applications.
18:10So, this is you know, create a nice front end and put in a stock.
18:15Oh, not slash.
18:17So, we can go ahead and do VOO.
18:19And this ticker agent is a bunch of our different managed agents that will go out and do specific tasks and look for different analysis on a stock.
18:37And again, this is kind of more of an illustration illustration that's like as you're building managed agents into your own apps, you might not want to have this kind of stream
18:45but just to kind of visualize what the actual agent is doing.
18:49It's reasoning through all the different steps in the interactions API.
18:53And then this managed agent is told to kind of generate a report based on all of the findings and reasoning it did in its sandbox.
19:00So, now if we go ahead and click on the report, we can see the managed agent outputted this kind of PDF file for us where we can see all of the research it did,
19:10see its analysis and key takeaways, as well as doing these charts so we can visualize.
19:15And it's also going to go and link out to kind of all of the resources um that it went ends actually using for this.
19:24So it actually went to the Vanguard Vanguard site, found the different documents, read them in, and then did analysis on it.
19:33And that is it on managed agents.
19:36Thank you very much.