We Gave an Agent Production Code Access and Then Tried to Sleep at Night — Moritz Johner, Form3

AI Engineer · 21 min · 258 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Thanks everyone for joining in.
00:14Um thanks for the great intro, by the way.
00:16Um So, yeah.
00:17My talk today is about um so titles we give in Asia production code access and then try to sleep at night.
00:23Um so, it's mostly around dependency dependency patching, which is probably the most glamorous problem in software engineering.
00:33I guess everyone did it here at some point in the past.
00:36Um it really is like vacuuming.
00:39Like everyone loves it, can't wait to do do it again next week.
00:42Now, at our scale, we have thousands of repositories and it really is a backlog that never empties and you close 10 issues today and you know next week 20 more will arrive and you have to deal with them.
00:54So, naturally you think, "Sure.
00:57Let's just automate it.
00:58Dependabot exists.
01:00Renovate exists.
01:01Problem solved, right?"
01:03It isn't.
01:04Cuz these tools were really built for a world where fixing a CVE means um um looking at the manifest in a repository and just now bumping a version to the next version and that's it.
01:17Um but our world is much more complicated than that, at least nowadays in our world since we have containers.
01:22So, the first problem is that the vulnerable thing isn't necessarily the thing that these tools can see.
01:28For instance, the CVE might live in an OS package that you use in your base image.
01:32It's not in your Dockerfile.
01:34It's just in the base image.
01:36Or if your Dockerfile pulls down a binary during the build process, it may see the URL, which might contain a version number, but it has no idea how to act on it.
01:45So, they can only look at the manifest, patch the manifest, but everything else is just simply invisible.
01:52Now, the second problem is patches never happen in isolation.
01:57So, when we look at the Go code base, sometimes you want to bump the Go runtime to the next minor version.
02:03When you do that, sometimes you might have to also bump the Go linter cuz the version are just intertwined.
02:08You have to bump both of them at at the same time.
02:11So, when you bump the linter, that sometimes introduces new linting rules, which then invalidates the code base, and you're left with this big mess.
02:19Now, these old tools, quote unquote old tools, Dependabot, well, they just bump the Go runtime version, create a PR, and walk away, and you're just left with this mess, and you need to figure out
02:30how to fix that.
02:36So, you don't really have a like a patching problem, you also have like a reasoning problem that you um need to address here.
02:44You really want to have a tool that looks at the CVE and understands the full surface of it.
02:48Where does the CVE actually live?
02:50And needs to figure out what else needs to move in order to um get to a green CI.
02:57So, that's basically what we did.
02:58Um we built Patch Pilot.
03:01We pushed to production, and eventually Infosec um came around the corner ask a very reasonable question, is this automation, or is it a supply chain incident waiting to happen?
03:12A useful coding agent is a supply chain actor, whether you plan for that or not.
03:17That's the thesis of this talk, basically.
03:20It's not agents are dangerous, or agents are fine.
03:22It's the moment where you give an agent um production credentials in order to like be useful, it really becomes a supply chain actor, just like an engineer in your department.
03:33And you should apply similar or the same guardrails uh for the agent just as you do for engineers.
03:40Now, this is a case study.
03:41We built this, we ran it in production.
03:43Infosec pushed back in several places.
03:45They were right.
03:46Um I just want to present to you what we came up with in the end.
03:49Um it's nothing new, it's nothing fancy.
03:51Um I just want to share share what we did and then hopefully have a discussion afterwards to see how we all figured this out together.
04:01Now, patch pilot has two layers.
04:03One, it's a simple go application that is deterministic.
04:06It's boring.
04:07It just does orchestration.
04:09It discovers vulnerable artifacts.
04:11So, artifacts that's our OCI images.
04:14That's what we have in production.
04:15That's what we ship to our customers and that's the thing that we really care about.
04:20So, the go app discovers those, scans the images, and figures out which image is built by which repository.
04:28Because we need that link in order to later then work on the repositories.
04:34And this deterministic part is very boring on purpose.
04:37It's very simple.
04:38And inside that, we spawn agents.
04:41Now, these agents are there for the for the reasoning.
04:44For instance, when we see a CI failure, we just tell the agent, "Look, that's a CI failure.
04:47Please figure out what else to do in this case."
04:51It could be that CI failed because a previous change that we did.
04:55It could also be a flaky CI.
04:58It could be an infrastructure failure, just a simple timeout that just, you know, caused the CI failure.
05:02And then then we would just need to retry and do nothing else in that case.
05:08Now, let's have a look at it how it looks end-to-end.
05:12So, first and foremost, a little bit of color coding.
05:14I hope you can see this.
05:15So, the green bits, that's our deterministic part, our go application.
05:19And the yellow ones, that's the agent.
05:21Um So, first we start with the with the assessment.
05:26That's just a simple manifest, a simple JSON file which contains a list of images and the CVEs that it found for these particular images.
05:33Um It will then moves on and downloads the repository it's supposed to work on, um clones it into a local directory and creates another directory inside it where it then shoves
05:44all the context for the agent inside there.
05:47That's a prompt with which has like 2,000 words or so.
05:50Um it's the assessment manifest, um a couple of other things for like communicating with the agent.
05:59And then simply we simply invoke the CVE remediation agent, which is like the first bit.
06:04Um the first agentic part that we invoke, which is supposed to do the the smallest effective change set.
06:09And it should only fix the CVEs which are tied to the findings.
06:12It shouldn't just, you know, bump the dependencies to the latest and greatest version.
06:16That's just that interest introduces unnecessary risk, which we want to avoid.
06:21So just bump to the version that actually fixes the CVE and um nothing more.
06:27And at the very end, the agent then of course should um just verify its own work.
06:32Should ensure that the Dockerfile is able to build, should um rescan the image that it has built, and all of that stuff.
06:40Um so the CVE remediation agent actually just modifies files on the file system.
06:44It doesn't commit, it doesn't push, it doesn't create a PR, it doesn't watch the CI itself.
06:49It just modifies files on the file system.
06:52And once it's done, it hands back control to the controller, to the deterministic bit, which then quickly vets the changes for, you know, some nonsense in there in there.
07:03For instance, like um empty files or some binaries that the agent might create because agents are really dumb.
07:10They mess up, you know, some bash commands that are piped together, which then could um create empty files.
07:17Or they just, you know, compile the application and leave the binary inside the repository, which is nonsense.
07:22So we got to take care of that.
07:24And then we commit it, we push it, we create a PR, um and then we watch CI.
07:30And once CI is green, cool, we just send over Slack notifications and we're done.
07:34Um in case there's a CI failure, we just move to the next agentic bit, which um should then just remediate the PR failure.
07:43To do that, we just give it all the context that it needs.
07:46So, CI logs, workflow metadata that we need, and everything else that is needed to work on this on the CI failure.
07:54And then we invoke it.
07:55And this PR remediation agent just simply should do the minimum effort fixes.
08:00It kind of LLMs LLMs kind of tend to just revert the previous changes that it did.
08:05So, we got to tell it to not do this.
08:08And again, it should just verify its own work.
08:10And this then goes on in loop, create and watch the PR, watch CI, fixes the CI failure, and so on and so forth, until we hit like a a maximum retry.
08:21And then we send that over to a to a human in case there's like the limit is reached.
08:27We then just simply tell the tell the human to take a look at this particular issue.
08:32Cuz then that needs manual remediation.
08:36There's one more thing that I want to share here, which is that at the end of every agent invocation, we ask the agent to do a very short and simple retrospective.
08:46What went well, what went wrong, what tools are missing, and what kind of context would help the next time it would be invoked.
08:52And this really helps us to understand what the agent is doing and what is what is missing out.
08:57Um Now, the observability for the agent in bits, that's still like an open issue and being built by the community at the moment.
09:04That's just how we deal dealt with this with this particular issue.
09:08Um So, yeah.
09:10That really helps us to understand this at scale because then we can just aggregate all this all that information across the PRs, condense it down, and then we can see
09:18um what we need to fix.
09:19We usually saw like two kinds of issues.
09:21One is an infrastructure-related issue.
09:24It could be um network failures, or that the agent didn't have enough permissions in order to like clone a repository, um or it could be some complexity issue related with the repository.
09:35Some repositories are just hard to deal with and hard to reason about without having the the context that the agent needs.
09:43Then we just need to either modify the system prompt of the agent or we then have repository-specific instructions um that we then just feed um to our agent to work on this issue.
09:55I brought a screenshot here with me.
09:58Um I had to redact a lot of stuff cuz it's an actual like production kind of screenshot and you know, in finance you don't want to share this kind of information, but um yeah, that's what I what I got out.
10:09So, here at the top where the arrow is, that's a very simple bit.
10:12We just bump the the base image that we're using from the Go line that simply a dependabot can do.
10:17That's boring.
10:18At the bottom we can see that the agent uh updated libcrypto3 and libssl3.
10:24So, um yeah.
10:26I also have like a short diff um that I want to quickly showcase what the agent did.
10:32Um you can see here on the left-hand side that it that these um bottom two Can I just go there?
10:37Here.
10:39That we have these two packages here that it actually removed in the process because these packages um have been pinned previously.
10:44It then just removed those because they have been fixed by the actual base image.
10:48So, to be honest, if I were tasked with this kind of task, I probably would have missed that.
10:54So, I'm like glad that the agent um fixed that and is a good good engineer here.
10:59So, and that's it.
11:00So, we started with a bunch of CVEs.
11:02We scanned artifacts, um remediated remediated this the CVEs and the findings, and now we have a proper working um um PR that then just needs to be reviewed by a human and approved and merged.
11:15So, to make that work, um we gave Patch Pilot a couple of things.
11:19We gave it GitHub access, um read and write access to clone the repository, to commit, and to push changes, to open up PR, to download the CI logs, and trigger CI.
11:29We also gave it OCR registry credentials in order to list images in the in the registry to figure out, you know, what images are available, which are latest.
11:37Um we gave it tools like a go run time, python run time, a bunch of linters, static libs, a bash shell, network access, and all of that.
11:47Um So, this is what we gave the whole application.
11:51But again, we had like two layers, the deterministic part and the agentic part.
11:54And we also applied the split for the credentials also for the two different layers.
11:59So, here's the thing about the capability list.
12:02The dangerous ones, the get up right access, um and trigger UCI is something that we did not give the agent.
12:07Instead, we pushed um that functionality out to the deterministic part cuz that's the thing that we can reason about and we can rely on that um you know, it just does these kind of kind of actions and we do not give um
12:22the agent these kinds of credentials cuz that then fundamentally fundamentally limits the blast radius of when in case the agent gets um prompt injected prompt injected.
12:33So, that boundary really matters for the prompt injection case because you probably didn't see it in the screenshot earlier, but there was like 70,000 lines of code that were changed in that small PR.
12:43Um that's really like a lot of changes that come in just by bumping a couple of dependencies.
12:48Um and the attack surface is really really really wide.
12:51Um So, what we did to mitigate that, I guess like prompt injection itself isn't solved and we cannot really solve it.
12:58All we can do is just to limit the blast radius in case that happens.
13:02Um what we did is to do a little bit of prompt steering because we know what kind of directories or files contain untrusted um un untrusted um information, untrusted context.
13:13We just tell the agent, look, the vendor directory, just don't trust that.
13:18Or the CI logs which live in that file or in that subdirectory, you know, be sure that you don't, you know, be an idiot.
13:24Um there's another thing that we did um, which worked quite well, which is that we, um, we essentially implemented an end-to-end test where we created a repository and sent patch patch pilot added to just work on it.
13:40Um, I guess that's what people call eval's today and then we just, um, ensure that it isn't, um, prompt injected.
13:48So, we have like a deprecated function in there in the in our crafted repository which tries to recruit the agent to do some malicious stuff or we have a migration guide where it deprecated function, um,
13:59points at and all of that.
14:01So, we try to like, you know, remediate these kinds of, um, issues that we know about.
14:06But still there um, unknown, um, injection vectors which we aren't aware of yet.
14:11Um, so that's why, you know, we still have to pray a little bit.
14:14But at least we don't like build the whole system on on hope.
14:20Now, that's the bit that kept me awake at night.
14:23Um, Now, sandboxes look great on a slide.
14:28You just draw a box, put the agent in it and you feel secure, right?
14:32So, the problem is that at some point the agent really wants to, um, verify its own work.
14:38When it works with Docker files, it wants to build a Docker container.
14:41It might also want to run a Docker container in order to figure out what package versions are available and so on.
14:47So, naturally you give it that Docker socket.
14:50At that point, it's more or less game over for you, um, because the agent can then simply just spawn a privileged container, escape out of it and then, you know, read environment variables of other processes,
15:01read the memory of other processes, can plant SSH keys, it's game over for you essentially at this point.
15:09We run it like that in production at some point.
15:11Um, it didn't feel good.
15:12We moved off of that, um, and we evaluated all the other obvious options in like the Linux sphere Linux bubble when it comes to like sandboxing.
15:21There's a lot of technologies out there like landlock, bubble wrap, seccomp, u notify, fence, and a lot of functions that we have for unprivileged Docker builds, um Kaniko, uh buildkit, and and what else.
15:32But, they don't really compose well with containers, and none of them really can contain a Docker socket or a Docker um demon that runs on a host.
15:42So, let me share a design that we came up with, which is still like in its in its infancy.
15:46Um It's the same pattern.
15:49You just draw a box, but instead of calling it a sandbox, it's just a micro VM.
15:53In our case, we're using um using [clears throat] Firecracker to have a like a proper isolation mechanism.
15:58And then we put the agent in, we put the Docker socket in, and then the Docker socket is powered by its own kernel in this case.
16:06Um which is good because that really solves the issue with with the um Docker daemon or the agent trying to escape the sandbox.
16:14In this case, everything is just contained within that micro VM, which is good.
16:19There's another thing that is um worth mentioning in this case.
16:23Again, we have this two-tier architecture, this two layers, the agentic bit and the deterministic bit.
16:28And we also want to apply network policies to these two worlds separately.
16:34For the deterministic bit, we just know what kind of um um network access it requires in order to function properly.
16:40It needs GitHub access and what else.
16:42For the agent, we don't really know in advance.
16:44It kind of depends off on the kind of repository or language um it works on.
16:50When it uses Java, it needs like a completely different ecosystem it works with compared to Python or Go.
16:55Um so, that's why we really want to need to apply different network policies.
16:59Um so, that's what you can do.
17:01You can just set up a DNS and TCP forwarder inside that micro VM.
17:05But, other than that, just, you know, cut it completely off from the host system.
17:08And then everything has to flow through this um through this Vsock, which then um hands all the network packets to a host process which then applies the network policies to
17:19um to all the outgoing packets.
17:21And then you can apply, you know, rules based on host names, based on target ports, side ranges, whatever you like.
17:29If you really want to go wild, you can just, you know, set up a custom CA, plant it into the micro VM, and then do all sorts of like TLS man-in-the-middle control.
17:38Which is really hard to do, especially when it comes to the Docker socket, but um we're not going to dive into this one yet.
17:45So, my take on agent sandboxing is this, that the existing agent that we have today with Codex and Cloud, they come with their own sandbox, but in my opinion, it's worthless, especially when you give it um
17:59a a Docker socket access.
18:01It's just really yeah, it's not going to be able to contain that.
18:07Other agents like Open Code or Pi, they just don't have a sandbox because it's not implemented um or it's just not there by design.
18:15So, my take is that you really have to create a environment for an agent where you can just put it in and give it everything that it needs, um so it has like all the freedom to work on a particular issue that it's supposed to do.
18:28Um but there's still a gap.
18:33So, the diagram that I've just shown you isn't something unique.
18:36It's just a VM with a little bit of plumbing in there.
18:39You can just wipe code, it's not not a big deal, honestly.
18:42Um you can even go down sta- downstairs to the vendor booths and just, you know, talk to the vendors there.
18:47You have like a lot of sandbox as a service that you can easily consume.
18:52Some of which really lack in functionality, especially in that regards to like containing a Docker socket or having a proper network access controls.
19:00That's still lacking, but it differs from vendor to vendor.
19:04I also recently saw the um micro sandbox project, um which has now run for, I think, like three or four months or so, maybe even longer, um which really is would be my choice if I would build patch pilot today
19:17again because it comes with all the batteries included, network access controls, um all of that.
19:23So, keep that in mind if you really want to build this.
19:25Um it's an open source project.
19:28It has It has a community.
19:29It's currently funded by YC.
19:31Um so, we really don't really know how it how it goes in the next couple of months.
19:35We're going to see um if they're able to gather enough traction to to keep the open source project afloat.
19:40Let's see.
19:42So, the gap isn't the tool doesn't exist.
19:45All the tools do exist, but most of them are still in the beta phase.
19:49And there's still a big gap to, you know, getting actual enterprise traction and getting all the features, all the bells and whistles that you need in order to deploy it into a proper enterprise environment.
19:59And I think that's something that we need to work on.
20:01Sure, now we got a sandbox.
20:03Cool.
20:03That's nice, but we still need like a lot of orchestration on top of it, which is something that you can take from a vendor.
20:10There are also like a couple of open source project at the moment which are working on this.
20:13There's the Kubernetes um agent sandbox uh special interest group that's working on this.
20:18There's um open sandbox which also working on this to integrate these kind of things.
20:22So, um keep that in mind if you really want to, you know, contain an agent in a production environment.
20:26It's not there yet.
20:27It's beta.
20:28It's very early in the in the phase.
20:30We're going to see how it goes.
20:35All right.
20:35If you take one thing from this, um the blast radius of an agent is an architecture decision.
20:40We didn't um give the agent the credential that it needs um in order to, you know, trigger CI or push to GitHub or open a PR.
20:48That's just simply not needed.
20:50You can just push that functionality in functionality into a deterministic layer.
20:54Um that kind of really limits the blast radius of an agent.
20:59So, that choice, what's that what's deterministic and what's agentic, that really is, you know, your security model in this case.
21:07All right.
21:08Um that was it for me.
21:11I just have like one last word, which is just an invitation to the community just to talk about the things that I just mentioned.
21:17Um I think tomorrow there's a a dedicated uh sandbox panel where a lot of talks are just around the sandbox thingy.
21:23And surely I want to know from you guys how you run agents in production just to learn and to have a discussion about that in order to, you know, bring the community on and just just figure it out um all together.
21:34Thank you.
21:36[applause]
21:54[music]