WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=LqLoYksJ6do

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/LqLoYksJ6do.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.040 --> 00:00:14.480
Thanks everyone for joining in.

s3
00:00:14.480 --> 00:00:16.120
Um thanks for the great intro, by the way.

s4
00:00:16.120 --> 00:00:17.920
Um So, yeah.

s5
00:00:17.920 --> 00:00:23.960
My talk today is about um so titles we give in Asia production code access and then try to sleep at night.

s6
00:00:23.960 --> 00:00:33.840
Um so, it's mostly around dependency dependency patching, which is probably the most glamorous problem in software engineering.

s7
00:00:33.840 --> 00:00:36.720
I guess everyone did it here at some point in the past.

s8
00:00:36.720 --> 00:00:39.040
Um it really is like vacuuming.

s9
00:00:39.040 --> 00:00:42.320
Like everyone loves it, can't wait to do do it again next week.

s10
00:00:42.320 --> 00:00:54.640
Now, at our scale, we have thousands of repositories and it really is a backlog that never empties and you close 10 issues today and you know next week 20 more will arrive and you have to deal with them.

s11
00:00:54.640 --> 00:00:57.440
So, naturally you think, "Sure.

s12
00:00:57.440 --> 00:00:58.880
Let's just automate it.

s13
00:00:58.880 --> 00:01:00.640
Dependabot exists.

s14
00:01:00.640 --> 00:01:01.360
Renovate exists.

s15
00:01:01.360 --> 00:01:03.600
Problem solved, right?"

s16
00:01:03.600 --> 00:01:04.760
It isn't.

s17
00:01:04.760 --> 00:01:17.080
Cuz these tools were really built for a world where fixing a CVE means um um looking at the manifest in a repository and just now bumping a version to the next version and that's it.

s18
00:01:17.080 --> 00:01:22.600
Um but our world is much more complicated than that, at least nowadays in our world since we have containers.

s19
00:01:22.600 --> 00:01:28.640
So, the first problem is that the vulnerable thing isn't necessarily the thing that these tools can see.

s20
00:01:28.640 --> 00:01:32.880
For instance, the CVE might live in an OS package that you use in your base image.

s21
00:01:32.880 --> 00:01:34.320
It's not in your Dockerfile.

s22
00:01:34.320 --> 00:01:36.520
It's just in the base image.

s23
00:01:36.520 --> 00:01:45.440
Or if your Dockerfile pulls down a binary during the build process, it may see the URL, which might contain a version number, but it has no idea how to act on it.

s24
00:01:45.440 --> 00:01:52.680
So, they can only look at the manifest, patch the manifest, but everything else is just simply invisible.

s25
00:01:52.840 --> 00:01:57.120
Now, the second problem is patches never happen in isolation.

s26
00:01:57.200 --> 00:02:03.000
So, when we look at the Go code base, sometimes you want to bump the Go runtime to the next minor version.

s27
00:02:03.000 --> 00:02:08.399
When you do that, sometimes you might have to also bump the Go linter cuz the version are just intertwined.

s28
00:02:08.399 --> 00:02:11.280
You have to bump both of them at at the same time.

s29
00:02:11.280 --> 00:02:19.280
So, when you bump the linter, that sometimes introduces new linting rules, which then invalidates the code base, and you're left with this big mess.

s30
00:02:19.280 --> 00:02:30.360
Now, these old tools, quote unquote old tools, Dependabot, well, they just bump the Go runtime version, create a PR, and walk away, and you're just left with this mess, and you need to figure out

s31
00:02:30.360 --> 00:02:32.800
how to fix that.

s32
00:02:36.760 --> 00:02:44.120
So, you don't really have a like a patching problem, you also have like a reasoning problem that you um need to address here.

s33
00:02:44.120 --> 00:02:48.040
You really want to have a tool that looks at the CVE and understands the full surface of it.

s34
00:02:48.040 --> 00:02:50.120
Where does the CVE actually live?

s35
00:02:50.120 --> 00:02:56.520
And needs to figure out what else needs to move in order to um get to a green CI.

s36
00:02:57.200 --> 00:02:58.840
So, that's basically what we did.

s37
00:02:58.840 --> 00:03:01.840
Um we built Patch Pilot.

s38
00:03:01.840 --> 00:03:11.800
We pushed to production, and eventually Infosec um came around the corner ask a very reasonable question, is this automation, or is it a supply chain incident waiting to happen?

s39
00:03:12.880 --> 00:03:17.280
A useful coding agent is a supply chain actor, whether you plan for that or not.

s40
00:03:17.280 --> 00:03:20.160
That's the thesis of this talk, basically.

s41
00:03:20.160 --> 00:03:22.640
It's not agents are dangerous, or agents are fine.

s42
00:03:22.640 --> 00:03:33.400
It's the moment where you give an agent um production credentials in order to like be useful, it really becomes a supply chain actor, just like an engineer in your department.

s43
00:03:33.400 --> 00:03:40.880
And you should apply similar or the same guardrails uh for the agent just as you do for engineers.

s44
00:03:40.959 --> 00:03:41.880
Now, this is a case study.

s45
00:03:41.880 --> 00:03:43.840
We built this, we ran it in production.

s46
00:03:43.840 --> 00:03:45.480
Infosec pushed back in several places.

s47
00:03:45.480 --> 00:03:46.680
They were right.

s48
00:03:46.680 --> 00:03:49.320
Um I just want to present to you what we came up with in the end.

s49
00:03:49.320 --> 00:03:51.920
Um it's nothing new, it's nothing fancy.

s50
00:03:51.920 --> 00:04:00.360
Um I just want to share share what we did and then hopefully have a discussion afterwards to see how we all figured this out together.

s51
00:04:01.640 --> 00:04:03.320
Now, patch pilot has two layers.

s52
00:04:03.320 --> 00:04:06.920
One, it's a simple go application that is deterministic.

s53
00:04:06.920 --> 00:04:07.440
It's boring.

s54
00:04:07.440 --> 00:04:09.480
It just does orchestration.

s55
00:04:09.480 --> 00:04:11.840
It discovers vulnerable artifacts.

s56
00:04:11.840 --> 00:04:14.360
So, artifacts that's our OCI images.

s57
00:04:14.360 --> 00:04:15.560
That's what we have in production.

s58
00:04:15.560 --> 00:04:20.799
That's what we ship to our customers and that's the thing that we really care about.

s59
00:04:20.799 --> 00:04:28.480
So, the go app discovers those, scans the images, and figures out which image is built by which repository.

s60
00:04:28.480 --> 00:04:33.440
Because we need that link in order to later then work on the repositories.

s61
00:04:34.600 --> 00:04:37.000
And this deterministic part is very boring on purpose.

s62
00:04:37.000 --> 00:04:38.280
It's very simple.

s63
00:04:38.280 --> 00:04:41.400
And inside that, we spawn agents.

s64
00:04:41.400 --> 00:04:44.000
Now, these agents are there for the for the reasoning.

s65
00:04:44.000 --> 00:04:47.960
For instance, when we see a CI failure, we just tell the agent, "Look, that's a CI failure.

s66
00:04:47.960 --> 00:04:51.000
Please figure out what else to do in this case."

s67
00:04:51.000 --> 00:04:55.280
It could be that CI failed because a previous change that we did.

s68
00:04:55.280 --> 00:04:58.120
It could also be a flaky CI.

s69
00:04:58.120 --> 00:05:02.880
It could be an infrastructure failure, just a simple timeout that just, you know, caused the CI failure.

s70
00:05:02.880 --> 00:05:08.000
And then then we would just need to retry and do nothing else in that case.

s71
00:05:08.000 --> 00:05:12.040
Now, let's have a look at it how it looks end-to-end.

s72
00:05:12.160 --> 00:05:14.400
So, first and foremost, a little bit of color coding.

s73
00:05:14.400 --> 00:05:15.160
I hope you can see this.

s74
00:05:15.160 --> 00:05:19.080
So, the green bits, that's our deterministic part, our go application.

s75
00:05:19.080 --> 00:05:21.600
And the yellow ones, that's the agent.

s76
00:05:21.600 --> 00:05:26.040
Um So, first we start with the with the assessment.

s77
00:05:26.040 --> 00:05:33.120
That's just a simple manifest, a simple JSON file which contains a list of images and the CVEs that it found for these particular images.

s78
00:05:33.120 --> 00:05:44.520
Um It will then moves on and downloads the repository it's supposed to work on, um clones it into a local directory and creates another directory inside it where it then shoves

s79
00:05:44.520 --> 00:05:47.120
all the context for the agent inside there.

s80
00:05:47.120 --> 00:05:50.880
That's a prompt with which has like 2,000 words or so.

s81
00:05:50.880 --> 00:05:58.240
Um it's the assessment manifest, um a couple of other things for like communicating with the agent.

s82
00:05:59.720 --> 00:06:04.280
And then simply we simply invoke the CVE remediation agent, which is like the first bit.

s83
00:06:04.280 --> 00:06:09.960
Um the first agentic part that we invoke, which is supposed to do the the smallest effective change set.

s84
00:06:09.960 --> 00:06:12.840
And it should only fix the CVEs which are tied to the findings.

s85
00:06:12.840 --> 00:06:16.400
It shouldn't just, you know, bump the dependencies to the latest and greatest version.

s86
00:06:16.400 --> 00:06:21.360
That's just that interest introduces unnecessary risk, which we want to avoid.

s87
00:06:21.360 --> 00:06:27.720
So just bump to the version that actually fixes the CVE and um nothing more.

s88
00:06:27.880 --> 00:06:32.600
And at the very end, the agent then of course should um just verify its own work.

s89
00:06:32.600 --> 00:06:39.800
Should ensure that the Dockerfile is able to build, should um rescan the image that it has built, and all of that stuff.

s90
00:06:40.600 --> 00:06:44.560
Um so the CVE remediation agent actually just modifies files on the file system.

s91
00:06:44.560 --> 00:06:49.080
It doesn't commit, it doesn't push, it doesn't create a PR, it doesn't watch the CI itself.

s92
00:06:49.080 --> 00:06:52.160
It just modifies files on the file system.

s93
00:06:52.160 --> 00:07:03.600
And once it's done, it hands back control to the controller, to the deterministic bit, which then quickly vets the changes for, you know, some nonsense in there in there.

s94
00:07:03.600 --> 00:07:10.680
For instance, like um empty files or some binaries that the agent might create because agents are really dumb.

s95
00:07:10.680 --> 00:07:17.840
They mess up, you know, some bash commands that are piped together, which then could um create empty files.

s96
00:07:17.840 --> 00:07:22.520
Or they just, you know, compile the application and leave the binary inside the repository, which is nonsense.

s97
00:07:22.520 --> 00:07:24.160
So we got to take care of that.

s98
00:07:24.160 --> 00:07:29.800
And then we commit it, we push it, we create a PR, um and then we watch CI.

s99
00:07:30.040 --> 00:07:34.280
And once CI is green, cool, we just send over Slack notifications and we're done.

s100
00:07:34.280 --> 00:07:43.680
Um in case there's a CI failure, we just move to the next agentic bit, which um should then just remediate the PR failure.

s101
00:07:43.680 --> 00:07:46.040
To do that, we just give it all the context that it needs.

s102
00:07:46.040 --> 00:07:54.600
So, CI logs, workflow metadata that we need, and everything else that is needed to work on this on the CI failure.

s103
00:07:54.600 --> 00:07:55.640
And then we invoke it.

s104
00:07:55.640 --> 00:08:00.120
And this PR remediation agent just simply should do the minimum effort fixes.

s105
00:08:00.120 --> 00:08:05.680
It kind of LLMs LLMs kind of tend to just revert the previous changes that it did.

s106
00:08:05.680 --> 00:08:08.120
So, we got to tell it to not do this.

s107
00:08:08.120 --> 00:08:10.960
And again, it should just verify its own work.

s108
00:08:10.960 --> 00:08:21.800
And this then goes on in loop, create and watch the PR, watch CI, fixes the CI failure, and so on and so forth, until we hit like a a maximum retry.

s109
00:08:21.800 --> 00:08:27.600
And then we send that over to a to a human in case there's like the limit is reached.

s110
00:08:27.600 --> 00:08:32.560
We then just simply tell the tell the human to take a look at this particular issue.

s111
00:08:32.560 --> 00:08:35.560
Cuz then that needs manual remediation.

s112
00:08:36.360 --> 00:08:46.040
There's one more thing that I want to share here, which is that at the end of every agent invocation, we ask the agent to do a very short and simple retrospective.

s113
00:08:46.040 --> 00:08:52.200
What went well, what went wrong, what tools are missing, and what kind of context would help the next time it would be invoked.

s114
00:08:52.200 --> 00:08:57.080
And this really helps us to understand what the agent is doing and what is what is missing out.

s115
00:08:57.080 --> 00:09:04.280
Um Now, the observability for the agent in bits, that's still like an open issue and being built by the community at the moment.

s116
00:09:04.280 --> 00:09:08.720
That's just how we deal dealt with this with this particular issue.

s117
00:09:08.720 --> 00:09:10.640
Um So, yeah.

s118
00:09:10.640 --> 00:09:18.160
That really helps us to understand this at scale because then we can just aggregate all this all that information across the PRs, condense it down, and then we can see

s119
00:09:18.160 --> 00:09:19.680
um what we need to fix.

s120
00:09:19.680 --> 00:09:21.400
We usually saw like two kinds of issues.

s121
00:09:21.400 --> 00:09:24.480
One is an infrastructure-related issue.

s122
00:09:24.480 --> 00:09:35.760
It could be um network failures, or that the agent didn't have enough permissions in order to like clone a repository, um or it could be some complexity issue related with the repository.

s123
00:09:35.760 --> 00:09:43.800
Some repositories are just hard to deal with and hard to reason about without having the the context that the agent needs.

s124
00:09:43.800 --> 00:09:54.800
Then we just need to either modify the system prompt of the agent or we then have repository-specific instructions um that we then just feed um to our agent to work on this issue.

s125
00:09:55.920 --> 00:09:58.320
I brought a screenshot here with me.

s126
00:09:58.320 --> 00:10:09.120
Um I had to redact a lot of stuff cuz it's an actual like production kind of screenshot and you know, in finance you don't want to share this kind of information, but um yeah, that's what I what I got out.

s127
00:10:09.120 --> 00:10:12.520
So, here at the top where the arrow is, that's a very simple bit.

s128
00:10:12.520 --> 00:10:17.560
We just bump the the base image that we're using from the Go line that simply a dependabot can do.

s129
00:10:17.560 --> 00:10:18.520
That's boring.

s130
00:10:18.520 --> 00:10:24.600
At the bottom we can see that the agent uh updated libcrypto3 and libssl3.

s131
00:10:24.600 --> 00:10:26.960
So, um yeah.

s132
00:10:26.960 --> 00:10:32.080
I also have like a short diff um that I want to quickly showcase what the agent did.

s133
00:10:32.080 --> 00:10:37.960
Um you can see here on the left-hand side that it that these um bottom two Can I just go there?

s134
00:10:37.960 --> 00:10:39.160
Here.

s135
00:10:39.160 --> 00:10:44.960
That we have these two packages here that it actually removed in the process because these packages um have been pinned previously.

s136
00:10:44.960 --> 00:10:48.080
It then just removed those because they have been fixed by the actual base image.

s137
00:10:48.080 --> 00:10:54.280
So, to be honest, if I were tasked with this kind of task, I probably would have missed that.

s138
00:10:54.280 --> 00:10:59.680
So, I'm like glad that the agent um fixed that and is a good good engineer here.

s139
00:10:59.680 --> 00:11:00.280
So, and that's it.

s140
00:11:00.280 --> 00:11:02.040
So, we started with a bunch of CVEs.

s141
00:11:02.040 --> 00:11:15.600
We scanned artifacts, um remediated remediated this the CVEs and the findings, and now we have a proper working um um PR that then just needs to be reviewed by a human and approved and merged.

s142
00:11:15.600 --> 00:11:19.400
So, to make that work, um we gave Patch Pilot a couple of things.

s143
00:11:19.400 --> 00:11:29.800
We gave it GitHub access, um read and write access to clone the repository, to commit, and to push changes, to open up PR, to download the CI logs, and trigger CI.

s144
00:11:29.800 --> 00:11:37.200
We also gave it OCR registry credentials in order to list images in the in the registry to figure out, you know, what images are available, which are latest.

s145
00:11:37.200 --> 00:11:47.360
Um we gave it tools like a go run time, python run time, a bunch of linters, static libs, a bash shell, network access, and all of that.

s146
00:11:47.360 --> 00:11:51.120
Um So, this is what we gave the whole application.

s147
00:11:51.120 --> 00:11:54.560
But again, we had like two layers, the deterministic part and the agentic part.

s148
00:11:54.560 --> 00:11:59.240
And we also applied the split for the credentials also for the two different layers.

s149
00:11:59.240 --> 00:12:02.000
So, here's the thing about the capability list.

s150
00:12:02.000 --> 00:12:07.920
The dangerous ones, the get up right access, um and trigger UCI is something that we did not give the agent.

s151
00:12:07.920 --> 00:12:22.000
Instead, we pushed um that functionality out to the deterministic part cuz that's the thing that we can reason about and we can rely on that um you know, it just does these kind of kind of actions and we do not give um

s152
00:12:22.000 --> 00:12:33.720
the agent these kinds of credentials cuz that then fundamentally fundamentally limits the blast radius of when in case the agent gets um prompt injected prompt injected.

s153
00:12:33.960 --> 00:12:43.160
So, that boundary really matters for the prompt injection case because you probably didn't see it in the screenshot earlier, but there was like 70,000 lines of code that were changed in that small PR.

s154
00:12:43.160 --> 00:12:48.680
Um that's really like a lot of changes that come in just by bumping a couple of dependencies.

s155
00:12:48.920 --> 00:12:51.880
Um and the attack surface is really really really wide.

s156
00:12:51.880 --> 00:12:58.760
Um So, what we did to mitigate that, I guess like prompt injection itself isn't solved and we cannot really solve it.

s157
00:12:58.760 --> 00:13:02.440
All we can do is just to limit the blast radius in case that happens.

s158
00:13:02.440 --> 00:13:13.760
Um what we did is to do a little bit of prompt steering because we know what kind of directories or files contain untrusted um un untrusted um information, untrusted context.

s159
00:13:13.760 --> 00:13:18.120
We just tell the agent, look, the vendor directory, just don't trust that.

s160
00:13:18.120 --> 00:13:24.839
Or the CI logs which live in that file or in that subdirectory, you know, be sure that you don't, you know, be an idiot.

s161
00:13:24.839 --> 00:13:40.200
Um there's another thing that we did um, which worked quite well, which is that we, um, we essentially implemented an end-to-end test where we created a repository and sent patch patch pilot added to just work on it.

s162
00:13:40.200 --> 00:13:48.960
Um, I guess that's what people call eval's today and then we just, um, ensure that it isn't, um, prompt injected.

s163
00:13:48.960 --> 00:13:59.880
So, we have like a deprecated function in there in the in our crafted repository which tries to recruit the agent to do some malicious stuff or we have a migration guide where it deprecated function, um,

s164
00:13:59.880 --> 00:14:01.400
points at and all of that.

s165
00:14:01.400 --> 00:14:06.360
So, we try to like, you know, remediate these kinds of, um, issues that we know about.

s166
00:14:06.360 --> 00:14:11.120
But still there um, unknown, um, injection vectors which we aren't aware of yet.

s167
00:14:11.120 --> 00:14:14.840
Um, so that's why, you know, we still have to pray a little bit.

s168
00:14:14.840 --> 00:14:18.880
But at least we don't like build the whole system on on hope.

s169
00:14:20.840 --> 00:14:23.680
Now, that's the bit that kept me awake at night.

s170
00:14:23.680 --> 00:14:28.040
Um, Now, sandboxes look great on a slide.

s171
00:14:28.040 --> 00:14:32.600
You just draw a box, put the agent in it and you feel secure, right?

s172
00:14:32.600 --> 00:14:38.560
So, the problem is that at some point the agent really wants to, um, verify its own work.

s173
00:14:38.560 --> 00:14:41.400
When it works with Docker files, it wants to build a Docker container.

s174
00:14:41.400 --> 00:14:47.920
It might also want to run a Docker container in order to figure out what package versions are available and so on.

s175
00:14:47.920 --> 00:14:50.560
So, naturally you give it that Docker socket.

s176
00:14:50.560 --> 00:15:01.880
At that point, it's more or less game over for you, um, because the agent can then simply just spawn a privileged container, escape out of it and then, you know, read environment variables of other processes,

s177
00:15:01.880 --> 00:15:08.480
read the memory of other processes, can plant SSH keys, it's game over for you essentially at this point.

s178
00:15:09.120 --> 00:15:11.400
We run it like that in production at some point.

s179
00:15:11.400 --> 00:15:12.360
Um, it didn't feel good.

s180
00:15:12.360 --> 00:15:21.000
We moved off of that, um, and we evaluated all the other obvious options in like the Linux sphere Linux bubble when it comes to like sandboxing.

s181
00:15:21.000 --> 00:15:32.520
There's a lot of technologies out there like landlock, bubble wrap, seccomp, u notify, fence, and a lot of functions that we have for unprivileged Docker builds, um Kaniko, uh buildkit, and and what else.

s182
00:15:32.520 --> 00:15:42.120
But, they don't really compose well with containers, and none of them really can contain a Docker socket or a Docker um demon that runs on a host.

s183
00:15:42.800 --> 00:15:46.600
So, let me share a design that we came up with, which is still like in its in its infancy.

s184
00:15:46.600 --> 00:15:49.480
Um It's the same pattern.

s185
00:15:49.480 --> 00:15:53.680
You just draw a box, but instead of calling it a sandbox, it's just a micro VM.

s186
00:15:53.680 --> 00:15:58.480
In our case, we're using um using [clears throat] Firecracker to have a like a proper isolation mechanism.

s187
00:15:58.480 --> 00:16:06.400
And then we put the agent in, we put the Docker socket in, and then the Docker socket is powered by its own kernel in this case.

s188
00:16:06.400 --> 00:16:14.240
Um which is good because that really solves the issue with with the um Docker daemon or the agent trying to escape the sandbox.

s189
00:16:14.240 --> 00:16:19.800
In this case, everything is just contained within that micro VM, which is good.

s190
00:16:19.800 --> 00:16:23.440
There's another thing that is um worth mentioning in this case.

s191
00:16:23.440 --> 00:16:28.880
Again, we have this two-tier architecture, this two layers, the agentic bit and the deterministic bit.

s192
00:16:28.880 --> 00:16:34.080
And we also want to apply network policies to these two worlds separately.

s193
00:16:34.400 --> 00:16:40.280
For the deterministic bit, we just know what kind of um um network access it requires in order to function properly.

s194
00:16:40.280 --> 00:16:42.839
It needs GitHub access and what else.

s195
00:16:42.839 --> 00:16:44.600
For the agent, we don't really know in advance.

s196
00:16:44.600 --> 00:16:50.120
It kind of depends off on the kind of repository or language um it works on.

s197
00:16:50.120 --> 00:16:55.240
When it uses Java, it needs like a completely different ecosystem it works with compared to Python or Go.

s198
00:16:55.240 --> 00:16:59.480
Um so, that's why we really want to need to apply different network policies.

s199
00:16:59.480 --> 00:17:01.120
Um so, that's what you can do.

s200
00:17:01.120 --> 00:17:05.240
You can just set up a DNS and TCP forwarder inside that micro VM.

s201
00:17:05.240 --> 00:17:08.760
But, other than that, just, you know, cut it completely off from the host system.

s202
00:17:08.760 --> 00:17:19.720
And then everything has to flow through this um through this Vsock, which then um hands all the network packets to a host process which then applies the network policies to

s203
00:17:19.720 --> 00:17:21.400
um to all the outgoing packets.

s204
00:17:21.400 --> 00:17:28.680
And then you can apply, you know, rules based on host names, based on target ports, side ranges, whatever you like.

s205
00:17:29.000 --> 00:17:38.000
If you really want to go wild, you can just, you know, set up a custom CA, plant it into the micro VM, and then do all sorts of like TLS man-in-the-middle control.

s206
00:17:38.000 --> 00:17:45.560
Which is really hard to do, especially when it comes to the Docker socket, but um we're not going to dive into this one yet.

s207
00:17:45.840 --> 00:17:59.560
So, my take on agent sandboxing is this, that the existing agent that we have today with Codex and Cloud, they come with their own sandbox, but in my opinion, it's worthless, especially when you give it um

s208
00:17:59.560 --> 00:18:01.640
a a Docker socket access.

s209
00:18:01.640 --> 00:18:07.320
It's just really yeah, it's not going to be able to contain that.

s210
00:18:07.320 --> 00:18:15.760
Other agents like Open Code or Pi, they just don't have a sandbox because it's not implemented um or it's just not there by design.

s211
00:18:15.760 --> 00:18:28.000
So, my take is that you really have to create a environment for an agent where you can just put it in and give it everything that it needs, um so it has like all the freedom to work on a particular issue that it's supposed to do.

s212
00:18:28.000 --> 00:18:33.120
Um but there's still a gap.

s213
00:18:33.120 --> 00:18:36.880
So, the diagram that I've just shown you isn't something unique.

s214
00:18:36.880 --> 00:18:39.240
It's just a VM with a little bit of plumbing in there.

s215
00:18:39.240 --> 00:18:42.280
You can just wipe code, it's not not a big deal, honestly.

s216
00:18:42.280 --> 00:18:47.840
Um you can even go down sta- downstairs to the vendor booths and just, you know, talk to the vendors there.

s217
00:18:47.840 --> 00:18:52.200
You have like a lot of sandbox as a service that you can easily consume.

s218
00:18:52.200 --> 00:19:00.440
Some of which really lack in functionality, especially in that regards to like containing a Docker socket or having a proper network access controls.

s219
00:19:00.440 --> 00:19:04.040
That's still lacking, but it differs from vendor to vendor.

s220
00:19:04.040 --> 00:19:17.920
I also recently saw the um micro sandbox project, um which has now run for, I think, like three or four months or so, maybe even longer, um which really is would be my choice if I would build patch pilot today

s221
00:19:17.920 --> 00:19:23.440
again because it comes with all the batteries included, network access controls, um all of that.

s222
00:19:23.440 --> 00:19:25.840
So, keep that in mind if you really want to build this.

s223
00:19:25.840 --> 00:19:28.040
Um it's an open source project.

s224
00:19:28.040 --> 00:19:29.560
It has It has a community.

s225
00:19:29.560 --> 00:19:31.160
It's currently funded by YC.

s226
00:19:31.160 --> 00:19:35.400
Um so, we really don't really know how it how it goes in the next couple of months.

s227
00:19:35.400 --> 00:19:40.840
We're going to see um if they're able to gather enough traction to to keep the open source project afloat.

s228
00:19:40.840 --> 00:19:42.560
Let's see.

s229
00:19:42.560 --> 00:19:45.160
So, the gap isn't the tool doesn't exist.

s230
00:19:45.160 --> 00:19:49.760
All the tools do exist, but most of them are still in the beta phase.

s231
00:19:49.760 --> 00:19:59.680
And there's still a big gap to, you know, getting actual enterprise traction and getting all the features, all the bells and whistles that you need in order to deploy it into a proper enterprise environment.

s232
00:19:59.680 --> 00:20:01.960
And I think that's something that we need to work on.

s233
00:20:01.960 --> 00:20:03.240
Sure, now we got a sandbox.

s234
00:20:03.240 --> 00:20:03.880
Cool.

s235
00:20:03.880 --> 00:20:10.320
That's nice, but we still need like a lot of orchestration on top of it, which is something that you can take from a vendor.

s236
00:20:10.320 --> 00:20:13.880
There are also like a couple of open source project at the moment which are working on this.

s237
00:20:13.880 --> 00:20:18.880
There's the Kubernetes um agent sandbox uh special interest group that's working on this.

s238
00:20:18.880 --> 00:20:22.280
There's um open sandbox which also working on this to integrate these kind of things.

s239
00:20:22.280 --> 00:20:26.680
So, um keep that in mind if you really want to, you know, contain an agent in a production environment.

s240
00:20:26.680 --> 00:20:27.800
It's not there yet.

s241
00:20:27.800 --> 00:20:28.440
It's beta.

s242
00:20:28.440 --> 00:20:30.800
It's very early in the in the phase.

s243
00:20:30.800 --> 00:20:33.320
We're going to see how it goes.

s244
00:20:35.160 --> 00:20:35.760
All right.

s245
00:20:35.760 --> 00:20:40.800
If you take one thing from this, um the blast radius of an agent is an architecture decision.

s246
00:20:40.800 --> 00:20:48.640
We didn't um give the agent the credential that it needs um in order to, you know, trigger CI or push to GitHub or open a PR.

s247
00:20:48.640 --> 00:20:50.160
That's just simply not needed.

s248
00:20:50.160 --> 00:20:54.400
You can just push that functionality in functionality into a deterministic layer.

s249
00:20:54.400 --> 00:20:59.000
Um that kind of really limits the blast radius of an agent.

s250
00:20:59.040 --> 00:21:06.560
So, that choice, what's that what's deterministic and what's agentic, that really is, you know, your security model in this case.

s251
00:21:07.840 --> 00:21:08.920
All right.

s252
00:21:08.920 --> 00:21:11.200
Um that was it for me.

s253
00:21:11.200 --> 00:21:17.040
I just have like one last word, which is just an invitation to the community just to talk about the things that I just mentioned.

s254
00:21:17.040 --> 00:21:23.960
Um I think tomorrow there's a a dedicated uh sandbox panel where a lot of talks are just around the sandbox thingy.

s255
00:21:23.960 --> 00:21:34.120
And surely I want to know from you guys how you run agents in production just to learn and to have a discussion about that in order to, you know, bring the community on and just just figure it out um all together.

s256
00:21:34.120 --> 00:21:36.200
Thank you.

s257
00:21:36.259 --> 00:21:38.259
[applause]

s258
00:21:54.600 --> 00:21:56.600
[music]