Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI

AI Engineer · 14 min · 149 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Thanks for everyone for coming.
00:14So, my name is Mikail.
00:16I head up the open source group at Arise AI and we work on a fully open source observability and evaluation framework and we also work on our own agent.
00:29So, today we want I want to kind of walk you through what Code Mode is and our journey embracing Code Mode.
00:36So, if you look back in the last 30 years of software development, you could kind of segment software into two different categories.
00:44You could think of there's software built for software engineers and software for non-engineers.
00:49Take for example, you have you know, you're asked to get all the invoices out of your email inbox.
00:55If you're a programmer, you could write a script,
00:57[snorts]
00:57you can go in there, you can dynamically like see if there's an attachment, download it, put it into a folder.
01:03It might be a you know, 50 lines of code and you get this tool that you can use versus if you were a non-technical, you'd have to go into these inboxes,
01:12you have to manually download it.
01:14You're restricted by the interface of the software and you had to kind of buy that software.
01:20And the thing that's really interesting about things like Cloud Code is that that boundary is completely dissolving, right?
01:26Basically, anything you need, the coding agent can actually build that software for you dynamically.
01:32And the labs and the foundation model providers know this, right?
01:35If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do.
01:40And so, doesn't it make sense basically to align the agents you're building to be able to kind of build the software you need dynamically to dissolve that boundary.
01:52So, before I get to started in Code Mode, let's look at kind of what an agent looks like.
01:58So, hopefully you've built this uh uh kind of agent before.
02:04You give an agent kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools that get pushed in the context window, it reasons about it, and then it calls another tool
02:17that get pushed in the context window.
02:19And hopefully most of you have built an agent like this.
02:21Uh certainly if you have a certain set of tools, this can work pretty well.
02:26But, we're now in that age where we want a lot more.
02:29We want much more emergent behavior.
02:31We want to put a lot more context into the agent.
02:34We want to have longer horizons.
02:36And so, this is where you kind of get this bloat inside of the context.
02:40You'll see there's a lot of tools that get added into the context window that might not be relevant to the task.
02:46Um maybe the tool call is not responding with a lot of junk, or maybe needs to paginate through data.
02:53And so, you deal with this kind of context rot.
02:56You also deal with multi-turn, a lot of turns, as a result, and you're going to have to deal with compaction.
03:02So, luckily CodeMon solves a lot of these things.
03:07It's kind of an alternative architecture.
03:09Um instead of, let's say, just giving it a huge index or a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool.
03:23But, you don't give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flows.
03:30So, they can do it sequentially, they can do them in parallel, um can put conditionals around it.
03:36Um and the nice thing about this is you can call multiple tools in one turn, and also return only what's necessary to push back into the context window of the model.
03:47And so, to kind of summarize it, put those these two things hypotheticals side by side, traditional tool calling, you know, you accumulate context window blow every time you call a tool.
03:59That's obviously going to, you know, add to latency and cost.
04:03It's also going to increase increase turns.
04:05So, the time your user takes to get to value increases, right?
04:10So, has a lot of overhead.
04:12Um, on on the flip side, code mode potentially, when done well, can unlock a lot of abilities because it can call as many as it wants in a single turn
04:23and it only has to put in exactly what's necessary into the context window to solve its problem and you can also get a reduction in turn.
04:31So, that's sort of the the goal of of code mode.
04:36And so, let's just kind of go over some of the reasons why code mode works.
04:40Code mode works because one, you don't have to give it the giant catalog of tools up front.
04:46It can just pick and choose the tools that it needs and understand those tools and then execute just those.
04:52So, there's a progressive disclosure of tools.
04:54Uh, certainly there's things like tool search now also built into the foundation model providers.
04:59There's also the composability.
05:00There's a huge amount of benefit to not being able to being able to call tools in different sequences, either in sequence or parallel, fan out, chaining them together.
05:10You can create much more complicated and efficient things.
05:13If you've heard of obviously Claude's dynamic workflows is built on this composability but composability of sub agents.
05:20I already mentioned sort of token efficiency.
05:23You don't have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window.
05:30The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window.
05:38And the fourth one, which is the kind of the one of the major reasons we've embraced code mode, is this idea of emergent behavior, being able to build its own tools, right?
05:45So, for example, traditional with traditional tool calling, if I said for example like take all these records and remove personally identifiable information, it wouldn't be able to do that because it doesn't have a tool, or it might try to do it inside of its its context window, but then you're leaking credit card numbers
06:01in the context window, and that's not great.
06:03So, the emerging behavior of being able to create its own tools has a huge amount of benefit.
06:11So, here are some ways in which we've embraced Code mod.
06:15Uh so, we build our own agent uh and harness inside of Phoenix, our open source uh our observability platform, and we started with Bash.
06:25You know, Bash is a great tool of being able to compose together things like uh you know, said and grep to be able to uh compose together, you know, built-in Linux primitives with also our own CLI
06:38and be able to offload those into files for later use.
06:42We also added um you know, a Python's sandbox, and we hoisted all our rest CRUD endpoints.
06:49Those are like perform getting traces or uh manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things.
07:04And we also added one additional one, which is the ability to write SQL.
07:09And you know, text of SQL, you've obviously like heard like that probably many times before, but this also unlocks a lot of capabilities.
07:17The reason why is not everything fits into the tool category of just creating, updating, and deleting.
07:22Maybe you need to be able to do aggregate queries like tell me all the LM spans that have P95 latency of X, Y, and Z. You don't want to have to paginate
07:31through all your traces to do that.
07:33You want it the LM to be able to pinpoint find exactly what you're looking for.
07:39This is kind of a a neat and creative way we've also used code mode.
07:43So, we've actually also created a sort of a programming paradigm in our UI that's bound to our UI.
07:48So, we've been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end.
07:57And what's really cool about this is you kind of almost get this computer loose use like emergent behavior because we're able to put that programming paradigm in a web worker that has a two-way data binding
08:09directly into our browser and it can kind of do complex UI interactions for you and it's really pretty fun to to watch it uh cook.
08:19So, code mode is not uh obviously for free.
08:22Um you're dealing with LLM generated code.
08:25If you've been at to the the Anthropic talks, you certainly probably seen some of these.
08:30You're dealing with resources.
08:31So, if you're working with hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around.
08:39Networking is a huge one.
08:40Maybe you want it to be able to go out to the web, but you never know what's out in the web.
08:44And so, you have to carefully think about network egress.
08:48Uh secrets is a hot topic.
08:51I think every sandbox provider is trying to come up with a different vaulting strategy.
08:56But, let's say you want to use Jev inside of your sandbox like Pi uh durable just shipped, you know, that's a crazy cool idea uh to be able to make fast decisions inside of a sandbox,
09:07but you have to make sure that those secrets don't come back into the context window.
09:12And then the other one is permissions.
09:13Uh there's some very interesting research around this as well, but typically permissions are overbroad.
09:19And so, maybe in the first five sets the permissions are correct, but in the sixth operation it is not.
09:25Or maybe in composition uh the permissions are not.
09:28And so, you have to kind of think of guard railing some of these things if that is necessary.
09:35There's also operational complexity.
09:38Uh certainly sandboxes don't boot uh all the time perfectly.
09:42And so you have to you know be able to deal with failures.
09:45They're harder to debug because now you're debugging code and you're not debug debugging individual tools.
09:51And tool parameter accuracy becomes a much bigger issue because the tool parameter now is code.
09:57Uh so what if something fails in the middle?
09:59Uh tool discovery is also still an issue.
10:04Previously if you did have a tool, the LM would up front know it, but now it might have to go through a handshake to be able to understand which tools
10:12are in its catalog of things to be able to do.
10:15And then this is sort of a big one, you know, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into harnesses.
10:24And now you have a double sandboxing uh problem.
10:27So sandboxing, you know, is going to return stuff out as serialized JSON and then it needs to be parsed and then put into another code mode.
10:34And so there's this like double duty that ends up happening that also can be on operational complexity.
10:42These are sort of open problems.
10:44Uh they're not like blockers, but durability is a problem.
10:48So let's say you build a giant piece of code that needs to execute and it breaks in the middle.
10:54You can't resume from that step.
10:56Um if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you're using.
11:05So JavaScript is something LLMs are really good at doing, but maybe not great for math.
11:10Um For example, uh code mode can be great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right?
11:22Because you don't want to serialize the whole binary, the whole image.
11:25That's going to end up in the the context window.
11:27So you might have to do some sort of signed image handoff.
11:31And then the other one which is like the Frontier models are amazing at code mode, but if you're trying to do cost saving and you want to swap out a smaller model, its capabilities might not just quite um get there.
11:44So, how do we at Arize evaluate code mode?
11:48Um we use our own We dogfood our own platform, which is called Arize Phoenix, um and we run our agents through various permutations through something called the Harbor framework,
12:00which basically puts the agents into uh sandbox environments and runs them on clear instructions with network access turned off, and we run this multiple times in trials, so we eliminate a lot of the flake.
12:13And we build evals in the form of Harbor verifiers to make sure that the agent's doing the correct thing.
12:20And then, Harbor's sort of rooted in this RL framework.
12:23We give it a a one or a zero to indicate whether or not performs the task, and we test it across many different permutations.
12:32So, this is kind of why you want to test like this, because as I mentioned, you can get a lot of flake.
12:38Um sometimes it might work where you're dealing with an indeterminate system.
12:42It might take different trajectories uh depending how you do it, and we uh track pass at K of each individual task that we think it should pass.
12:52So, you know, how does this work?
12:55Uh this is kind of our results from uh running this Harbor run.
12:59You can see pre uh code mode and post code mode, we, you know, decreased the amount of tokens by 43%.
13:07The cost went down by 31%, and you know, there was some overhead with the sandbox, but the latency stayed pretty constant.
13:12So, we knew that across these multiple surfaces, whether or not it was cloud code or MCP or the CLI, we had accomplished something that uh was definitely better than it was before.
13:23How do we use this data?
13:25It's really uh a huge unlock if you have you both capture the trajectory and the evals.
13:30As you've seen with a lot of the other talks, this gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb.
13:40So, maybe, you know, first time we introduce a new tool, sometimes actually the efficiency goes down.
13:45And so, we can tell a coding agent why in these specific scenarios, what parts of the instruction are are guiding it in the wrong direction and we're able to hill climb these
13:54evals to be able to make our agents better.
13:58So, if you don't take anything other than this away, hopefully this will sum it up.
14:05We really like code mode at RI because of the fewer turns you get out of code mode.
14:10We want you to be able to bring any model for us.
14:13So, we want it to be cheaper and we're really excited about the emergent behavior and the long horizon tasks that our agents can do.
14:20And so, I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it.
14:29And that's it.
14:30Thank you.