WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=k72eLcYpVys

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/k72eLcYpVys.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.800 --> 00:00:14.000
Hello everyone.

s3
00:00:14.000 --> 00:00:16.799
I think we can get started.

s4
00:00:17.359 --> 00:00:22.960
So, a few months ago, I did something that was um a bit slightly absurd.

s5
00:00:22.960 --> 00:00:24.640
I took a coding agent.

s6
00:00:24.640 --> 00:00:35.600
I pointed it at my team's uh cloud sandbox hosting environment and it said try to break it and I let that run overnight.

s7
00:00:35.600 --> 00:00:39.360
And this talk is a bit about what we found.

s8
00:00:39.600 --> 00:00:40.960
Uh my name is Kevin.

s9
00:00:40.960 --> 00:00:43.920
I work at Amazon Bedrock Agent Core.

s10
00:00:43.920 --> 00:00:47.520
It's a sandbox hosting environment or platform.

s11
00:00:47.520 --> 00:00:48.559
And we do more than that.

s12
00:00:48.559 --> 00:01:00.480
We provide agents, identity, memory, um, runtime environments, sandboxes, gateways, anything you want to use to build and host your agents on agents on the cloud.

s13
00:01:01.680 --> 00:01:14.000
So the way this started was basically with me taking a hard look at how um customers use our platform and increasingly I think a lot of us in the industry see that

s14
00:01:14.000 --> 00:01:20.080
uh a lot of platform components on the internet are just agents and increasingly they're also sandbox hosted.

s15
00:01:20.080 --> 00:01:28.080
I think today's talks will maybe walk you through all the architectures that can exist in cloud hosted uh sandbox environments.

s16
00:01:28.080 --> 00:01:38.479
And so this kind of got me thinking how can I perhaps emulate simulate the customer and also test the service the same way agents use our platform.

s17
00:01:39.600 --> 00:01:52.479
So the first thing I think we we set out to do was create an abstraction or interface that allows coding agents as well as agent applications to interact with our service

s18
00:01:52.479 --> 00:01:54.079
uh in a way that's pretty native to them.

s19
00:01:54.079 --> 00:01:59.280
And today we know that Frontier models know the MCP protocol pretty well.

s20
00:01:59.280 --> 00:02:08.640
So we spent a day or two just creating an MCP server to expose access to our environment and platform um just so it can get started pretty quickly.

s21
00:02:08.640 --> 00:02:10.800
The second piece was polishing that.

s22
00:02:10.800 --> 00:02:18.560
Uh we spent some time in finding the tools that can be made accessible and uh smoke testing some runs and and things like that.

s23
00:02:18.560 --> 00:02:29.280
So some of the tests we ran just by hand was uh for example write a Python code file and execute it or take a Python file and uh convert into Java.

s24
00:02:29.280 --> 00:02:35.040
And then we realized we wanted to just expand the test domains that we we were going to set the coding agents to do.

s25
00:02:35.040 --> 00:02:39.120
So we asked to just plainly write a thousand tests across 10 domains.

s26
00:02:39.120 --> 00:02:40.959
And a thousand tests here was pretty arbitrary.

s27
00:02:40.959 --> 00:02:44.160
It was just to cast a wide net and see what caught back.

s28
00:02:44.160 --> 00:02:50.160
This took a bit of polishing because we knew that a thousand tests would take a few hours and was going to be a little tokenheavy.

s29
00:02:50.160 --> 00:02:58.080
So we um did things like credential management credential management injection um what happens when a test fails.

s30
00:02:58.080 --> 00:02:58.959
Is it a service issue?

s31
00:02:58.959 --> 00:03:00.720
Is it a MCP layer issue?

s32
00:03:00.720 --> 00:03:05.840
Uh those are the kind of things we we let the agent um figure out.

s33
00:03:06.239 --> 00:03:14.560
So the architecture is honestly very simple but it's also very powerful and I think a lot of us have probably seen this in our day-to-day uh development.

s34
00:03:14.560 --> 00:03:17.360
So we have our coding agent on top left.

s35
00:03:17.360 --> 00:03:25.680
We have an MCP server that basically wraps our cloud sandbox environment and all the findings get sent back to us.

s36
00:03:25.680 --> 00:03:37.200
We store them in a SQLite database just because as some of you might have seen u agents are pretty prone to write markdown files but that doesn't really scale when you have a thousand tests that you want to run

s37
00:03:37.200 --> 00:03:47.120
and this allowed us to tap into concurrency or you can have multiple sub aents running locally and running and executing their tests and storing those results uh locally.

s38
00:03:47.920 --> 00:03:57.120
Now, a thousand tests I think is kind of easy to just say out loud, but really every single dot on the screen kind of represents a workflow that a customer would actually run.

s39
00:03:57.120 --> 00:04:00.799
So, we did things like testing the sandbox uh life cycle.

s40
00:04:00.799 --> 00:04:03.840
How reliable is it when you put it under stress?

s41
00:04:03.840 --> 00:04:10.640
Um things like file system IO, what happens when you upload a file and download it from the sandbox?

s42
00:04:10.640 --> 00:04:13.760
Uh tested performance.

s43
00:04:13.760 --> 00:04:23.600
We tested some software development life cycles like uh for example writing Python packages, Java packages um and the like.

s44
00:04:24.080 --> 00:04:30.240
So to give you a perspective of what one of these dots could represent, it could be an end toend machine learning training uh program.

s45
00:04:30.240 --> 00:04:34.000
It could be generating a PDF report.

s46
00:04:34.000 --> 00:04:38.880
Uh it could also just be um as simple as opening a web page.

s47
00:04:38.880 --> 00:04:52.880
And we re quickly realized that our platform which at the time was hosting two uh major applications, one was a code sandbox, the other was a cloud browser would have different tests that mattered differently to the sandboxes that they were hosted in.

s48
00:04:52.880 --> 00:05:01.680
So for example, here in the browser, we asked the agent to create a bunch of uh tabs and try to scrape them all and summarize them, maybe take some screenshots,

s49
00:05:01.680 --> 00:05:05.440
download them, maybe fill out a form, automate a login workflow.

s50
00:05:05.440 --> 00:05:15.440
But the point here is these sandbox hosted applications uh tend to go through different workflows that demand different requirements from the sandbox.

s51
00:05:16.240 --> 00:05:27.440
So we ran through the tests and I want to talk about just three failures that I think um showcase some of the patterns we saw uh from these uh sandbox hosted applications.

s52
00:05:27.440 --> 00:05:37.280
So a simple one was we uploaded an image, we downloaded it and we wanted to see if the same thing that we put up uh we got back when we asked to download it and lo and behold

s53
00:05:37.280 --> 00:05:38.560
that was not the case.

s54
00:05:38.560 --> 00:05:44.160
Uh we deep dived a bit into it and we found that the encoding layer at the MCP layer was incorrect.

s55
00:05:44.160 --> 00:05:46.960
The second one was a bit interesting too which was a no bite.

s56
00:05:46.960 --> 00:05:48.880
It just threw off off our tests.

s57
00:05:48.880 --> 00:05:51.680
Um a no bite can be a zero.

s58
00:05:51.680 --> 00:05:58.720
It could also be a marker in some coding languages that just mean that means that the string is is uh complete.

s59
00:05:58.720 --> 00:06:10.720
Now an interesting one that was particular the sandbox layer is one where we actually wrote files to a specific directory in the sandbox and when we try to get it back it just wouldn't be available.

s60
00:06:11.600 --> 00:06:16.960
So now we started to segregate the patterns uh and failures into two layers.

s61
00:06:16.960 --> 00:06:20.400
One was at the interface and the other one was at the sandbox layer.

s62
00:06:20.400 --> 00:06:35.199
And it's important I think to draw the distinction because uh when we start to host applications in sandboxes we they deviate a little bit from the typical API service pattern where you have a long running service that has exposes an API

s63
00:06:35.199 --> 00:06:41.680
uh and you you know send a request you get an output that's reasonably um shaped and you you kind of know what your backend service does.

s64
00:06:41.680 --> 00:06:52.960
But obviously in today's world we we no longer have as many long running applications rather we have specialized agents that are uh sandbox and environment we expose access to them through a gateway

s65
00:06:52.960 --> 00:06:58.479
and the demands that they have on their let's say computer environment are going to differ greatly.

s66
00:06:58.479 --> 00:07:07.199
We know that and this is an experience I think a lot of us can relate to which is you are developing something locally and you just start to see your agent

s67
00:07:07.199 --> 00:07:21.120
uh either a curo cloud codeex whatever it might be do funny things like it might uh download a package you might not have expected or it might translate a CSV file to JSON to PDF for whatever reason

s68
00:07:21.120 --> 00:07:26.720
and all of these workflows that your agents will want to do are going to demand uh different things from your environment.

s69
00:07:26.720 --> 00:07:35.520
So things for example like how much memory or CPU your sandbox has the permissions that are available to your processes and file systems.

s70
00:07:35.520 --> 00:07:40.720
I think in a lot of the talks we'll see today there are a lot of different architectures that your sandboxes can have.

s71
00:07:40.720 --> 00:07:49.599
And the wider point here I think is all these constraints have pretty massive implications for your agents ability to solve these problems.

s72
00:07:51.440 --> 00:07:59.280
And just to kind of visualize that, these are some of the faults that we saw when we compared two different applications hosted in sandboxes.

s73
00:07:59.280 --> 00:08:03.680
So at the top we have the code sandbox and the bottom we have the cloud browser.

s74
00:08:03.680 --> 00:08:11.759
At the bottom right we have the failure modes that are highlighted in red which signify how many times the sandbox actually uh perished.

s75
00:08:11.759 --> 00:08:16.720
In this case, because cloud browsers, we know that whenever we open up tabs, they're pretty memory heavy.

s76
00:08:16.720 --> 00:08:20.639
And so, a lot of our tests fail just due to memory constraints.

s77
00:08:20.639 --> 00:08:25.840
And that wasn't really a fault we saw in the uh code interpret environment.

s78
00:08:25.840 --> 00:08:32.240
And when we map them out a little bit differently, we start to see that the failure modes coales around sandbox primitives.

s79
00:08:32.240 --> 00:08:43.919
So we know some of the components around file systems tend to be file system permissions uh mounted file systems uh permissions to read, write, execute files, um whether or not the file system

s80
00:08:43.919 --> 00:08:46.720
uh is persistent or uh ephemeral.

s81
00:08:46.720 --> 00:08:50.320
Um and the tests were pretty unforgiving.

s82
00:08:50.320 --> 00:09:00.320
Um for example, in the bottom right we have uh the browser tests just starving the the sessions and sandboxes out of memory and signaling that as a complete failure.

s83
00:09:00.320 --> 00:09:09.680
And I think the wider point here is as we host these applications in sandboxes and we expect our customers to be able to build uh pretty arbitrary workloads in these sandboxes,

s84
00:09:09.680 --> 00:09:17.839
we have to be pretty careful that the sandbox constraints have massive implications as to those agents ability to complete those problems.

s85
00:09:18.720 --> 00:09:27.519
Now if we look a little bit closely and this slide is pretty heavy on the second column we have a lot of the primitives used to compose what a sandbox container

s86
00:09:27.519 --> 00:09:28.720
um might look like.

s87
00:09:28.720 --> 00:09:43.519
So we can go as deep as kernel capabilities, things like kernel interceptors, um these things called extended BP BPF programs which allow you to run kernel modules and intercept programs at the at the Python layer.

s88
00:09:43.519 --> 00:09:55.680
Um those might uh control how your agent actually can make some egress calls, maybe make a internal API call to um your internal company services.

s89
00:09:59.120 --> 00:10:11.440
If we diagram this out, this is kind of what a very high level overview of how the components in the sandbox might uh affect your um workload's ability to do its job.

s90
00:10:11.440 --> 00:10:21.440
So here for example, I think earlier in our talks, we spoke about uh security secure compute which controls the system control programs that your applications might want to run.

s91
00:10:21.440 --> 00:10:25.680
So for example, say I have a machine learning package that needs to spawn a process.

s92
00:10:25.680 --> 00:10:29.519
Does my process or container even have the ability to start a new process?

s93
00:10:29.519 --> 00:10:44.000
Um perhaps I have a code interpreter uh task that needs to download a Python rep Python package into a very specific module for uh the Python kernel to be able to pick it up or

s94
00:10:44.000 --> 00:10:49.920
are the Rust binaries in the right location for when the coding agent is able to um run or execute a program.

s95
00:10:49.920 --> 00:10:58.000
All these very very small things tend to add up into your agents ability to uh execute the business problem.

s96
00:10:58.959 --> 00:11:06.959
Now just taking a step back I think this was kind of interesting uh which was the idea of like using an NLM to generate basically unit test.

s97
00:11:06.959 --> 00:11:07.920
I think we've all done it.

s98
00:11:07.920 --> 00:11:11.519
We I don't know if anyone writes unit tests by hand anymore.

s99
00:11:11.519 --> 00:11:23.279
Uh and similarly if we go up up a layer integration tests uh we we we can anticipate agents to be pretty good at understanding the problem space and writing tests for them.

s100
00:11:23.279 --> 00:11:37.519
But as we move towards hosting more and more applications which are just agents uh in sandboxes, we start to see the cracks in the model which is um these very low-level primitives and controls

s101
00:11:37.519 --> 00:11:42.399
um might actually hamper your agents ability to do its state of goal.

s102
00:11:42.399 --> 00:11:56.160
Um, and what we were hoping to relay here is a a very strong way to actually surface these failure modes that are that are pretty innocuous to us is just through fuzzy testing your agent sandbox asking it to complete

s103
00:11:56.160 --> 00:12:01.200
perhaps in this case a thousand workflows and seeing what results you get.

s104
00:12:01.440 --> 00:12:09.760
And just to also make it a little bit more tangible, um, a lot of us here developers, we are very used to running coding agents locally.

s105
00:12:09.760 --> 00:12:13.120
You might have your, you know, your your codeex or your cloud running locally.

s106
00:12:13.120 --> 00:12:28.000
And as we expand and create more workspaces, you quickly put a constraint on your laptop and now, you know, your teams want to put these coding agents in remote sandboxes so you can scale how many GitHub PRs you you work through and the like.

s107
00:12:28.000 --> 00:12:37.760
But as you've probably seen, the tools that you have in your local workstation, your MCP tools, your locally running applications might not match what you have in a remote sandbox.

s108
00:12:38.399 --> 00:12:42.240
And just a small shout out to uh agent core.

s109
00:12:42.240 --> 00:12:50.959
This is actually one of the problems that we're very keen on because we recognize that customers want to scale their coding agents um and how customers are actually running these

s110
00:12:50.959 --> 00:12:52.399
uh software development life cycles.

s111
00:12:52.399 --> 00:13:00.560
So one of the releases we recently did was grant the ability to customers to SSH into their um sandbox just to see what's going on.

s112
00:13:00.560 --> 00:13:05.120
And of course it's not really a human who is SSH into your sandbox anymore.

s113
00:13:05.120 --> 00:13:08.240
or it's a coding agent who you're developing with uh as well.

s114
00:13:08.240 --> 00:13:21.120
But the idea is there uh bridging into the sandbox and poking around and seeing what problems are causing your coding agent or your application to not run as smoothly is a pretty um stated business problem for us

s115
00:13:21.120 --> 00:13:24.480
and one that we're trying to solve for customers.

s116
00:13:25.360 --> 00:13:34.399
And if you look a little bit uh if you look a little bit at academia uh one interesting example I think a lot of us probably care about is

s117
00:13:34.399 --> 00:13:44.000
whenever frontier models come out the first thing we look at is the sweet benchmark report which is okay this model got XYZ percentage on this three benchmark versus another

s118
00:13:44.000 --> 00:13:54.959
and there was a recent study made by some researchers at UC Berkeley where they found that given the right environment controls and an exploit agent they actually beat the three benchmarks like reaching up to 100%.

s119
00:13:54.959 --> 00:14:05.920
And in inversely uh there was another study where um researchers were trying to give a wide set of business problems to an agent with broken tools.

s120
00:14:05.920 --> 00:14:12.079
Uh and of course the effectiveness of that agent uh went down greatly.

s121
00:14:12.720 --> 00:14:24.959
So I think this is uh mostly where I want to aim to leave today which is as these applications that we build today are more and more agentic and

s122
00:14:24.959 --> 00:14:38.639
they're not constrained to a rest API service that you you build for customers but rather uh an arbitrary agent running inside of sandbox it's really really um uh challenging to find all the sharp edges where the sandbox primitives

s123
00:14:38.639 --> 00:14:43.760
uh lead to you know a less efficient uh ability for your agents to complete that problem.

s124
00:14:43.760 --> 00:14:54.399
So my challenge here for you today is uh if you're looking to host your agents on sandboxes, sure test it locally, but also fuzzy test it with a wide set of domain tests

s125
00:14:54.399 --> 00:14:56.560
um remotely.

s126
00:14:59.120 --> 00:15:01.680
Yeah, thank you.

s127
00:15:01.945 --> 00:15:03.945
[applause]

s128
00:15:16.355 --> 00:15:18.355
[music]