1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS

AI Engineer · 15 min · 128 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello everyone.
00:14I think we can get started.
00:17So, a few months ago, I did something that was um a bit slightly absurd.
00:22I took a coding agent.
00:24I pointed it at my team's uh cloud sandbox hosting environment and it said try to break it and I let that run overnight.
00:35And this talk is a bit about what we found.
00:39Uh my name is Kevin.
00:40I work at Amazon Bedrock Agent Core.
00:43It's a sandbox hosting environment or platform.
00:47And we do more than that.
00:48We provide agents, identity, memory, um, runtime environments, sandboxes, gateways, anything you want to use to build and host your agents on agents on the cloud.
01:01So the way this started was basically with me taking a hard look at how um customers use our platform and increasingly I think a lot of us in the industry see that
01:14uh a lot of platform components on the internet are just agents and increasingly they're also sandbox hosted.
01:20I think today's talks will maybe walk you through all the architectures that can exist in cloud hosted uh sandbox environments.
01:28And so this kind of got me thinking how can I perhaps emulate simulate the customer and also test the service the same way agents use our platform.
01:39So the first thing I think we we set out to do was create an abstraction or interface that allows coding agents as well as agent applications to interact with our service
01:52uh in a way that's pretty native to them.
01:54And today we know that Frontier models know the MCP protocol pretty well.
01:59So we spent a day or two just creating an MCP server to expose access to our environment and platform um just so it can get started pretty quickly.
02:08The second piece was polishing that.
02:10Uh we spent some time in finding the tools that can be made accessible and uh smoke testing some runs and and things like that.
02:18So some of the tests we ran just by hand was uh for example write a Python code file and execute it or take a Python file and uh convert into Java.
02:29And then we realized we wanted to just expand the test domains that we we were going to set the coding agents to do.
02:35So we asked to just plainly write a thousand tests across 10 domains.
02:39And a thousand tests here was pretty arbitrary.
02:40It was just to cast a wide net and see what caught back.
02:44This took a bit of polishing because we knew that a thousand tests would take a few hours and was going to be a little tokenheavy.
02:50So we um did things like credential management credential management injection um what happens when a test fails.
02:58Is it a service issue?
02:58Is it a MCP layer issue?
03:00Uh those are the kind of things we we let the agent um figure out.
03:06So the architecture is honestly very simple but it's also very powerful and I think a lot of us have probably seen this in our day-to-day uh development.
03:14So we have our coding agent on top left.
03:17We have an MCP server that basically wraps our cloud sandbox environment and all the findings get sent back to us.
03:25We store them in a SQLite database just because as some of you might have seen u agents are pretty prone to write markdown files but that doesn't really scale when you have a thousand tests that you want to run
03:37and this allowed us to tap into concurrency or you can have multiple sub aents running locally and running and executing their tests and storing those results uh locally.
03:47Now, a thousand tests I think is kind of easy to just say out loud, but really every single dot on the screen kind of represents a workflow that a customer would actually run.
03:57So, we did things like testing the sandbox uh life cycle.
04:00How reliable is it when you put it under stress?
04:03Um things like file system IO, what happens when you upload a file and download it from the sandbox?
04:10Uh tested performance.
04:13We tested some software development life cycles like uh for example writing Python packages, Java packages um and the like.
04:24So to give you a perspective of what one of these dots could represent, it could be an end toend machine learning training uh program.
04:30It could be generating a PDF report.
04:34Uh it could also just be um as simple as opening a web page.
04:38And we re quickly realized that our platform which at the time was hosting two uh major applications, one was a code sandbox, the other was a cloud browser would have different tests that mattered differently to the sandboxes that they were hosted in.
04:52So for example, here in the browser, we asked the agent to create a bunch of uh tabs and try to scrape them all and summarize them, maybe take some screenshots,
05:01download them, maybe fill out a form, automate a login workflow.
05:05But the point here is these sandbox hosted applications uh tend to go through different workflows that demand different requirements from the sandbox.
05:16So we ran through the tests and I want to talk about just three failures that I think um showcase some of the patterns we saw uh from these uh sandbox hosted applications.
05:27So a simple one was we uploaded an image, we downloaded it and we wanted to see if the same thing that we put up uh we got back when we asked to download it and lo and behold
05:37that was not the case.
05:38Uh we deep dived a bit into it and we found that the encoding layer at the MCP layer was incorrect.
05:44The second one was a bit interesting too which was a no bite.
05:46It just threw off off our tests.
05:48Um a no bite can be a zero.
05:51It could also be a marker in some coding languages that just mean that means that the string is is uh complete.
05:58Now an interesting one that was particular the sandbox layer is one where we actually wrote files to a specific directory in the sandbox and when we try to get it back it just wouldn't be available.
06:11So now we started to segregate the patterns uh and failures into two layers.
06:16One was at the interface and the other one was at the sandbox layer.
06:20And it's important I think to draw the distinction because uh when we start to host applications in sandboxes we they deviate a little bit from the typical API service pattern where you have a long running service that has exposes an API
06:35uh and you you know send a request you get an output that's reasonably um shaped and you you kind of know what your backend service does.
06:41But obviously in today's world we we no longer have as many long running applications rather we have specialized agents that are uh sandbox and environment we expose access to them through a gateway
06:52and the demands that they have on their let's say computer environment are going to differ greatly.
06:58We know that and this is an experience I think a lot of us can relate to which is you are developing something locally and you just start to see your agent
07:07uh either a curo cloud codeex whatever it might be do funny things like it might uh download a package you might not have expected or it might translate a CSV file to JSON to PDF for whatever reason
07:21and all of these workflows that your agents will want to do are going to demand uh different things from your environment.
07:26So things for example like how much memory or CPU your sandbox has the permissions that are available to your processes and file systems.
07:35I think in a lot of the talks we'll see today there are a lot of different architectures that your sandboxes can have.
07:40And the wider point here I think is all these constraints have pretty massive implications for your agents ability to solve these problems.
07:51And just to kind of visualize that, these are some of the faults that we saw when we compared two different applications hosted in sandboxes.
07:59So at the top we have the code sandbox and the bottom we have the cloud browser.
08:03At the bottom right we have the failure modes that are highlighted in red which signify how many times the sandbox actually uh perished.
08:11In this case, because cloud browsers, we know that whenever we open up tabs, they're pretty memory heavy.
08:16And so, a lot of our tests fail just due to memory constraints.
08:20And that wasn't really a fault we saw in the uh code interpret environment.
08:25And when we map them out a little bit differently, we start to see that the failure modes coales around sandbox primitives.
08:32So we know some of the components around file systems tend to be file system permissions uh mounted file systems uh permissions to read, write, execute files, um whether or not the file system
08:43uh is persistent or uh ephemeral.
08:46Um and the tests were pretty unforgiving.
08:50Um for example, in the bottom right we have uh the browser tests just starving the the sessions and sandboxes out of memory and signaling that as a complete failure.
09:00And I think the wider point here is as we host these applications in sandboxes and we expect our customers to be able to build uh pretty arbitrary workloads in these sandboxes,
09:09we have to be pretty careful that the sandbox constraints have massive implications as to those agents ability to complete those problems.
09:18Now if we look a little bit closely and this slide is pretty heavy on the second column we have a lot of the primitives used to compose what a sandbox container
09:27um might look like.
09:28So we can go as deep as kernel capabilities, things like kernel interceptors, um these things called extended BP BPF programs which allow you to run kernel modules and intercept programs at the at the Python layer.
09:43Um those might uh control how your agent actually can make some egress calls, maybe make a internal API call to um your internal company services.
09:59If we diagram this out, this is kind of what a very high level overview of how the components in the sandbox might uh affect your um workload's ability to do its job.
10:11So here for example, I think earlier in our talks, we spoke about uh security secure compute which controls the system control programs that your applications might want to run.
10:21So for example, say I have a machine learning package that needs to spawn a process.
10:25Does my process or container even have the ability to start a new process?
10:29Um perhaps I have a code interpreter uh task that needs to download a Python rep Python package into a very specific module for uh the Python kernel to be able to pick it up or
10:44are the Rust binaries in the right location for when the coding agent is able to um run or execute a program.
10:49All these very very small things tend to add up into your agents ability to uh execute the business problem.
10:58Now just taking a step back I think this was kind of interesting uh which was the idea of like using an NLM to generate basically unit test.
11:06I think we've all done it.
11:07We I don't know if anyone writes unit tests by hand anymore.
11:11Uh and similarly if we go up up a layer integration tests uh we we we can anticipate agents to be pretty good at understanding the problem space and writing tests for them.
11:23But as we move towards hosting more and more applications which are just agents uh in sandboxes, we start to see the cracks in the model which is um these very low-level primitives and controls
11:37um might actually hamper your agents ability to do its state of goal.
11:42Um, and what we were hoping to relay here is a a very strong way to actually surface these failure modes that are that are pretty innocuous to us is just through fuzzy testing your agent sandbox asking it to complete
11:56perhaps in this case a thousand workflows and seeing what results you get.
12:01And just to also make it a little bit more tangible, um, a lot of us here developers, we are very used to running coding agents locally.
12:09You might have your, you know, your your codeex or your cloud running locally.
12:13And as we expand and create more workspaces, you quickly put a constraint on your laptop and now, you know, your teams want to put these coding agents in remote sandboxes so you can scale how many GitHub PRs you you work through and the like.
12:28But as you've probably seen, the tools that you have in your local workstation, your MCP tools, your locally running applications might not match what you have in a remote sandbox.
12:38And just a small shout out to uh agent core.
12:42This is actually one of the problems that we're very keen on because we recognize that customers want to scale their coding agents um and how customers are actually running these
12:50uh software development life cycles.
12:52So one of the releases we recently did was grant the ability to customers to SSH into their um sandbox just to see what's going on.
13:00And of course it's not really a human who is SSH into your sandbox anymore.
13:05or it's a coding agent who you're developing with uh as well.
13:08But the idea is there uh bridging into the sandbox and poking around and seeing what problems are causing your coding agent or your application to not run as smoothly is a pretty um stated business problem for us
13:21and one that we're trying to solve for customers.
13:25And if you look a little bit uh if you look a little bit at academia uh one interesting example I think a lot of us probably care about is
13:34whenever frontier models come out the first thing we look at is the sweet benchmark report which is okay this model got XYZ percentage on this three benchmark versus another
13:44and there was a recent study made by some researchers at UC Berkeley where they found that given the right environment controls and an exploit agent they actually beat the three benchmarks like reaching up to 100%.
13:54And in inversely uh there was another study where um researchers were trying to give a wide set of business problems to an agent with broken tools.
14:05Uh and of course the effectiveness of that agent uh went down greatly.
14:12So I think this is uh mostly where I want to aim to leave today which is as these applications that we build today are more and more agentic and
14:24they're not constrained to a rest API service that you you build for customers but rather uh an arbitrary agent running inside of sandbox it's really really um uh challenging to find all the sharp edges where the sandbox primitives
14:38uh lead to you know a less efficient uh ability for your agents to complete that problem.
14:43So my challenge here for you today is uh if you're looking to host your agents on sandboxes, sure test it locally, but also fuzzy test it with a wide set of domain tests
14:54um remotely.
14:59Yeah, thank you.
15:01[applause]
15:16[music]