WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ZFxh7sqbUZo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ZFxh7sqbUZo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.880 --> 00:00:13.560
All right, everybody.

s2
00:00:13.560 --> 00:00:15.560
We're going to talk about hacking.

s3
00:00:15.560 --> 00:00:16.480
I love hacking.

s4
00:00:16.480 --> 00:00:20.960
We have a very small audience here, so I assume everyone here loves hacking as well.

s5
00:00:20.960 --> 00:00:25.040
So, I want to talk about designing reinforcement learning environments for cybersecurity tasks.

s6
00:00:25.040 --> 00:00:35.320
Essentially, we all want to teach computers to hack because well, we're pushing out programs faster than other ever and so we need to be able to check them at machine speeds in scale.

s7
00:00:35.320 --> 00:00:39.360
And this has been my research project for well over two decades.

s8
00:00:39.360 --> 00:00:40.600
My name is David Brumley.

s9
00:00:40.600 --> 00:00:46.440
I am a full professor at Carnegie Mellon University where I work on AI and cybersecurity.

s10
00:00:46.440 --> 00:00:51.640
And I'm also chief AI and science officer at Bugcrowd where I work on data partnerships.

s11
00:00:51.640 --> 00:01:02.480
So, before I talk about what we do and how we do it and why it's important to design cybersecurity tasks correctly for reinforcement learning environment, I want to start off with how humans learn because

s12
00:01:02.480 --> 00:01:05.239
I mean, I love teaching people to hack.

s13
00:01:05.239 --> 00:01:10.480
And I remember in particular a case where we run a hacking contest called picoCTF.

s14
00:01:10.480 --> 00:01:15.560
picoCTF has about a million high school kids every year play in this contest.

s15
00:01:15.560 --> 00:01:19.200
Um and so it's a a really fun way for people to get an intro to cybersecurity.

s16
00:01:19.200 --> 00:01:26.240
So, in 2016, a young uh person showed up on our scoreboard who was going by the hacker name fluorescence.

s17
00:01:26.240 --> 00:01:29.520
And typically, we know who is doing well in the contest.

s18
00:01:29.520 --> 00:01:36.600
It's kind of the typical suspects like a Palo Alto High School or uh some of the Washington D.C. high schools.

s19
00:01:36.600 --> 00:01:37.960
We know who's going to win the contest.

s20
00:01:37.960 --> 00:01:44.360
And so, this kind of independent starts showing up uh scoring on our scoreboard and we had no idea who it was.

s21
00:01:44.360 --> 00:01:45.360
So, we reach out.

s22
00:01:45.360 --> 00:01:51.960
It's actually a 17-year-old kid who found out about cybersecurity trying to get into it from math competitions.

s23
00:01:51.960 --> 00:01:54.920
He got bored with the math competitions and started doing them.

s24
00:01:54.920 --> 00:02:00.760
And very quickly, he ended up actually scoring second in picoCTF competing against all these high school kids.

s25
00:02:00.760 --> 00:02:03.880
And we asked actually "How did you learn this?"

s26
00:02:03.880 --> 00:02:07.040
And what he said really was uh germane to this task.

s27
00:02:07.040 --> 00:02:14.280
What I did is I looked at the cybersecurity task, and then I started Googling "What is the information I needed?"

s28
00:02:14.280 --> 00:02:17.800
I would read about it, I'd look at write-ups, and then I'd start emulating that.

s29
00:02:17.800 --> 00:02:20.840
And this kid actually ended up coming in second.

s30
00:02:20.840 --> 00:02:31.880
I recruited him to CMU, and he followed this methodology of studying write-ups and practicing cybersecurity on a graduated scale, easy problems first, and then slowly getting more difficult.

s31
00:02:31.880 --> 00:02:34.840
And he actually turned into what's called a Pwn2Own winner.

s32
00:02:34.840 --> 00:02:40.000
So, Pwn2Own, if you've never heard of it, is one of the more elite cybersecurity competitions.

s33
00:02:40.000 --> 00:02:43.959
This kid, just 2 years after he first learned cybersecurity, enters.

s34
00:02:43.959 --> 00:02:49.200
And uh if you read about it at the time, he was the first one to hack a Tesla.

s35
00:02:49.200 --> 00:02:55.040
So, he walked out of this contest with $375,000 in cash and a brand new Tesla.

s36
00:02:55.040 --> 00:03:04.760
The reason I tell this story is actually the way we teach AI uh frontier models to hack is the same way that we've been successful teaching high school students,

s37
00:03:04.760 --> 00:03:08.360
such as Richard Zhu, to become Pwn2Own winners.

s38
00:03:08.360 --> 00:03:16.600
My other students include people like George Hotz, who did the first iPhone jailbreak, and current Pwn2Own winners like Sang Heon Lee.

s39
00:03:16.600 --> 00:03:23.880
And so, what I want to talk about is how we teach reinforcement learning, and do it the same way that we've been teaching hacking for a while.

s40
00:03:23.880 --> 00:03:27.600
And it really breaks down into two different axes.

s41
00:03:27.600 --> 00:03:34.280
The first thing when designing these sorts of tasks for people is to look at target difficulty.

s42
00:03:34.280 --> 00:03:43.120
There's a spectrum of different challenges that you can look at from toy problems through CTF and synthetic problems, all the way up to hardened targets.

s43
00:03:43.120 --> 00:03:49.200
The second axis for teaching machines to hack is really looking at exploitation difficulty.

s44
00:03:49.200 --> 00:03:56.440
For example, when we look at a toy program, we may start looking at the sort of skills it needs to acquire to be able to hack that.

s45
00:03:56.440 --> 00:04:02.960
For example, if you have a toy program and it has a bug, can the LLM figure out where the bug is?

s46
00:04:02.960 --> 00:04:09.000
Can it then prove that it knows where it is by triggering a crash or some other fault in the program?

s47
00:04:09.000 --> 00:04:11.720
But of course, hacking is not just crashing a program.

s48
00:04:11.720 --> 00:04:13.480
We want to take control of that program.

s49
00:04:13.480 --> 00:04:15.640
That's the beautiful thing about hacking.

s50
00:04:15.640 --> 00:04:17.480
It's bending computers to our will.

s51
00:04:17.480 --> 00:04:21.120
It's what makes it unique in the sciences.

s52
00:04:21.120 --> 00:04:24.040
So you look at things like, "Hey, there's a flaw in that program.

s53
00:04:24.040 --> 00:04:27.280
Can I use that to do arbitrary read writes in memory?

s54
00:04:27.280 --> 00:04:31.520
Or even to do a full arbitrary code execution exploit?"

s55
00:04:31.520 --> 00:04:42.640
And so if you remember nothing else from this talk, it's really the way that we teach LLMs, whether it be frontier models like Anthropic or private models that you're tuning in your house,

s56
00:04:42.640 --> 00:04:52.320
you follow these two axes where you're trying to come up with a set of tasks that increase in target difficulty along one, and then you're teaching specific cybersecurity skills

s57
00:04:52.320 --> 00:04:54.560
on the second.

s58
00:04:55.640 --> 00:04:57.880
In other words, hacking is really a ladder.

s59
00:04:57.880 --> 00:05:03.680
And this is what actually matches cybersecurity so well to reinforcement learning.

s60
00:05:03.680 --> 00:05:09.520
We have a ladder of tasks and we typically end up with a good oracle for whether they can achieve that task.

s61
00:05:09.520 --> 00:05:14.240
And so you can start to measure whether your model is learning the right set of capabilities.

s62
00:05:14.240 --> 00:05:20.280
So this talk is really divided into three parts.

s63
00:05:22.120 --> 00:05:25.360
The first one is to talk about vulnerability discovery.

s64
00:05:25.360 --> 00:05:36.560
And when we talk about vulnerability discovery, what we're talking about is in the variety of different programs that you encounter in real life, how do you design oracles that are correct

s65
00:05:36.560 --> 00:05:41.040
for determining whether or not a model has successfully been able to detect that vulnerability?

s66
00:05:41.040 --> 00:05:55.360
And what's interesting is several of the cybersecurity benchmarks out there were amazing first-generation pieces of work, but they have a critical flaw where the model will actually stop learning after it finds the easiest vulnerability.

s67
00:05:55.360 --> 00:05:58.160
And that can prevent them from getting smarter.

s68
00:05:58.160 --> 00:06:04.280
The second is I want to talk about how we are designing benchmarks to measure this ability to do weaponization.

s69
00:06:04.480 --> 00:06:08.480
And this is really where we get into where does security differentiate from bug finding.

s70
00:06:08.480 --> 00:06:13.840
And we'll talk about how well LLMs do against what I would call hard targets.

s71
00:06:13.840 --> 00:06:20.520
A hard target one easy way to to look at it is how much would you pay for an exploit that a model could produce.

s72
00:06:20.520 --> 00:06:27.240
We know Richard Zhu fluorescence was paid $375,000 and got a brand new Tesla for one exploit.

s73
00:06:27.240 --> 00:06:29.880
Can models achieve that capabilities today?

s74
00:06:29.880 --> 00:06:36.520
And then I'm going to just summarize ways that if you're interested in this environment, we can connect and do more work together.

s75
00:06:36.520 --> 00:06:37.320
So, very simple talk.

s76
00:06:37.320 --> 00:06:45.320
So, let's talk about the first axis of discovery and where you really want to um learn what you're going to be measuring.

s77
00:06:45.320 --> 00:06:53.400
This is a key part in reinforcement learning where if you set up the wrong task objective, the LLM will learn it, but it'll learn the wrong thing.

s78
00:06:53.400 --> 00:06:55.240
So, some definitions to begin with.

s79
00:06:55.240 --> 00:06:57.640
Let's start defining the problem.

s80
00:06:57.640 --> 00:07:02.720
When we think about reinforcement learning or we talk about gyms, there's some key components in that.

s81
00:07:02.720 --> 00:07:06.919
There's of course other things, but the key components are you need a vulnerable application.

s82
00:07:06.919 --> 00:07:10.919
And we like to enclose these inside container environments so that they're reproducible.

s83
00:07:10.919 --> 00:07:22.040
We make sure that they run and that you don't have variations between for example, if I run a program on this version of Linux versus a different version of Linux, it actually may behave differently.

s84
00:07:22.040 --> 00:07:24.760
And so, you want to standardize that with a vulnerable program.

s85
00:07:24.760 --> 00:07:26.600
You need a grading oracle.

s86
00:07:26.600 --> 00:07:31.520
Now, one of the things I think the previous talk was talking about was LLM as a judge is a reasonable thing.

s87
00:07:31.520 --> 00:07:34.760
What we found in cybersecurity is that is flawed.

s88
00:07:34.760 --> 00:07:38.080
The LLMs will always say they were successful hacking.

s89
00:07:38.080 --> 00:07:44.200
And so, what you want to come up with is a deterministic grading oracle for each of the different levels you're getting at.

s90
00:07:44.200 --> 00:07:51.400
For example, if you're trying to teach it to just find a bugs, maybe this grading Oracle is was it able to trigger a crash.

s91
00:07:51.400 --> 00:07:53.160
We'll talk about that more in a second.

s92
00:07:53.160 --> 00:07:59.640
So, you have this reinforcement learning environment or this gym environment and of course you have your LLM and an orchestrator that's going to talk to it.

s93
00:07:59.640 --> 00:08:06.320
The way we set up our task is very simply, we expose through MCP a few key functions, a setup function.

s94
00:08:06.320 --> 00:08:09.600
So, the LLM will call setup, it returns the problem definition.

s95
00:08:09.600 --> 00:08:19.120
We give it standard tool calls such as read and write inside the container inside a sandbox inside the container, and then a grading Oracle at the very end.

s96
00:08:19.120 --> 00:08:28.480
And so, you end up with this vulnerable program in here, a grading Oracle, and I'm going to assume that you've already verified that there is at least one flaw in this program.

s97
00:08:28.480 --> 00:08:31.600
Maybe you yourself have figured out that it can crash.

s98
00:08:31.600 --> 00:08:36.400
Maybe you have downloaded it from a bug report and you've been able to reproduce that vulnerability.

s99
00:08:36.400 --> 00:08:40.080
We won't get into that, that's part of our sauce that we do at Bugcrowd.

s100
00:08:40.080 --> 00:08:48.520
But, once you do that, you have this package environment and then your task prompt is going to be something very simple like, "Dear LLM, can you find and exploit the vulnerability?"

s101
00:08:48.520 --> 00:08:51.839
Now, you don't want to just ask, "Can you find the vulnerability?"

s102
00:08:51.839 --> 00:08:57.400
because then you won't be able to distinguish between an LLM hallucination and a real vulnerability.

s103
00:08:57.400 --> 00:09:02.160
So, you almost always ask it to actually exploit the vulnerability.

s104
00:09:02.160 --> 00:09:07.000
And that exploit is going to be key to how we do reinforcement learning.

s105
00:09:07.000 --> 00:09:10.360
So, the LLM does some thinking and it comes up with an exploit.

s106
00:09:10.360 --> 00:09:16.680
For example, this very, very simple program, if you just give it enough A's, you'll trigger a crash.

s107
00:09:16.680 --> 00:09:21.480
So, that's the LLM's witness, the proof of vulnerability that it was able to find something.

s108
00:09:21.480 --> 00:09:25.080
You run that input through your grading Oracle.

s109
00:09:25.080 --> 00:09:28.400
The Oracle that determines did the program misbehave or not.

s110
00:09:28.400 --> 00:09:34.880
In this case, the program would simply crash and you farm out your rewards.

s111
00:09:35.040 --> 00:09:39.480
This is a very elegant way and actually this is the way we teach people to hack.

s112
00:09:39.480 --> 00:09:42.160
We set up a deterministic auto grader.

s113
00:09:42.160 --> 00:09:45.040
For example, in CTFs, it's cuz you capture the flag.

s114
00:09:45.040 --> 00:10:00.840
Within a cybersecurity environment like this, the level one maybe can it crash, all the way up to control flow hijack, where for example, you may ask the LLM, "Can you do something like launch a calculator, some external program you shouldn't be able to run, or do a reverse shell?"

s115
00:10:00.840 --> 00:10:02.560
So, that's the basic setup.

s116
00:10:02.560 --> 00:10:03.720
But, there's a problem with this.

s117
00:10:03.720 --> 00:10:10.720
This is the way if you go look at the existing benchmarks like uh Cybex or uh CyberGym, they set up the task.

s118
00:10:10.720 --> 00:10:12.520
But, there's actually a problem here.

s119
00:10:12.520 --> 00:10:17.320
And that's because there's an assumption that the program only has one vulnerability.

s120
00:10:17.320 --> 00:10:24.640
I don't know about you, but it's very rare to find a program for which you know there's only one vulnerability.

s121
00:10:24.640 --> 00:10:30.000
So, what happens if you have two vulnerabilities here?

s122
00:10:30.000 --> 00:10:34.520
This actually breaks a lot of assumptions in current evaluation environments.

s123
00:10:34.520 --> 00:10:38.280
You ask the same question, "Dear LLM, can you find and exploit the vulnerability?"

s124
00:10:38.280 --> 00:10:42.560
But, now the LLM has a lot of freedom to reward hack.

s125
00:10:42.560 --> 00:10:46.440
For example, which vulnerability should it find?

s126
00:10:46.440 --> 00:10:55.120
If you came in only knowing about the first vulnerability, but there's a second one you didn't know about, what do you do if the LLM thinks it found a second one?

s127
00:10:55.120 --> 00:10:57.760
Or suppose you know two.

s128
00:10:57.760 --> 00:11:12.920
What we found is on existing benchmarks with multiple pro- with a real OSS benchmarks, there are multiple vulnerabilities, the LLM will just continue to find the easiest vulnerability, and that really limits its trajectory as far as what it can learn.

s129
00:11:12.920 --> 00:11:19.920
And then you have a question, if it does find a vulnerability you did not know about, well, how do you score it, right?

s130
00:11:19.920 --> 00:11:24.400
You certainly don't want to give tasks that have no vulnerabilities because then you don't know if you're wasting your time.

s131
00:11:24.400 --> 00:11:28.440
But, what if the LLM finds an unknown vulnerability?

s132
00:11:28.440 --> 00:11:31.680
Here's where you can run into a catch-22.

s133
00:11:31.839 --> 00:11:36.280
What existing benchmarks do is they tell the LLM which bug.

s134
00:11:36.280 --> 00:11:48.880
For example, in many of the benchmarks out there like side bench, they will give a backtrace that says for example, I know the vulnerability is in this backtrace which identifies the vulnerable function.

s135
00:11:48.880 --> 00:11:55.480
But at that point, you're teaching the LLM, but you're pointing at it exactly the problem.

s136
00:11:55.480 --> 00:12:00.720
So the LLM no longer has to reason about the program and that will stunt its reasoning capability.

s137
00:12:00.720 --> 00:12:06.840
Essentially, if you're nudging it and saying here's the vulnerability, it's in this function, it doesn't have to do a lot.

s138
00:12:06.840 --> 00:12:12.880
In fact, it can often fit that entire function in its context window and it doesn't have to reason much.

s139
00:12:12.880 --> 00:12:21.320
The second problem though is if you don't tell the LLM which one and there's multiple vulnerabilities, it can always just then reward hack the easiest problem.

s140
00:12:21.320 --> 00:12:29.720
And we see this in every foundational LLM out there and we see it in as far as I can tell most of the benchmarks out there.

s141
00:12:29.720 --> 00:12:39.920
There will be multiple vulnerabilities, it will be graded, but because the grading is just checking for for example, a crash, it's not exploring the full state space and the LLM will just keep returning

s142
00:12:39.920 --> 00:12:41.320
the same one.

s143
00:12:41.320 --> 00:12:43.920
This is also a problem in some of the public competitions.

s144
00:12:43.920 --> 00:12:47.200
For example, we won something called the Cyber Grand Challenge from DARPA.

s145
00:12:47.200 --> 00:12:51.240
It was the first challenge from DARPA to show that fully autonomous cyber is capable.

s146
00:12:51.240 --> 00:12:56.560
50% of the hand curated challenges had unknown vulnerabilities.

s147
00:12:56.560 --> 00:13:10.720
This was DARPA spent $60 million designing a contest trying to come up with problems that were well defined and well scoped and they accidentally added additional bugs and 50% of those were ones that were actually exploited.

s148
00:13:10.720 --> 00:13:15.400
So this idea that we're just going to create synthetic problems with one bug doesn't work.

s149
00:13:15.400 --> 00:13:19.360
People have tried it, spent a lot of money, you always introduce new ones.

s150
00:13:19.360 --> 00:13:23.640
Second problem is second example I'd show is the AIxCC.

s151
00:13:23.640 --> 00:13:25.200
I designed the scoring algorithm for this.

s152
00:13:25.200 --> 00:13:31.840
This is again a very large DARPA program that ran last year in DEF CON where 18 of the bugs found were unintended ones.

s153
00:13:31.840 --> 00:13:37.680
And so the TLDR on here is you can't just say, "Well, we're going to hand curate an environment with just one vulnerability."

s154
00:13:37.680 --> 00:13:39.200
Experts have tried, it doesn't work.

s155
00:13:39.200 --> 00:13:42.240
You have to change the problem definition.

s156
00:13:42.280 --> 00:13:46.960
So we've been thinking about this and what we developed is a new way to test is called the audit task.

s157
00:13:46.960 --> 00:13:55.360
Again, suppose you have two different bugs, but you flip the question from just find a bug to find all vulnerabilities discovered.

s158
00:13:55.360 --> 00:14:01.120
At this point, the LLM has the freedom to find multiple bugs and submit multiple proofs of vulnerabilities.

s159
00:14:01.120 --> 00:14:06.560
And it may be full proofs of vulnerabilities for bugs you know about and bugs you don't.

s160
00:14:06.640 --> 00:14:10.200
You run all vulnerabilities through your oracle.

s161
00:14:10.200 --> 00:14:12.480
And this is where it's very important to have a deterministic grader.

s162
00:14:12.480 --> 00:14:17.760
So for here, for example, there's two vulnerabilities, it gives us two inputs that crash both vulnerabilities.

s163
00:14:17.760 --> 00:14:22.480
And part of this grader now have to uniquify them to show that two different vulnerabilities are triggered.

s164
00:14:22.480 --> 00:14:27.120
Now if we didn't know about vuln two, this also gives us the opportunity to increase our ground truth.

s165
00:14:27.120 --> 00:14:31.120
We haven't told the vuln the LLM that we don't know about something that it found.

s166
00:14:31.120 --> 00:14:33.760
It just gave us proof that it was able to find it.

s167
00:14:33.760 --> 00:14:43.800
So we can normalize the set of known vulnerabilities at that point to be something like D star and calculate the the precision and recall for the model across multiple vulnerabilities.

s168
00:14:43.800 --> 00:14:50.839
For example, recall is the number of known that it found over the total set and precision is the number of found over the submitted.

s169
00:14:50.839 --> 00:15:01.360
What this prevents the model from doing and it essentially balances is the ability for it to go find unknown vulnerabilities, but also prevents you from prevents the model from just spamming.

s170
00:15:01.360 --> 00:15:09.640
You don't want it to give you a bunch of things that aren't vulnerabilities, like for example, giving us POV in four that doesn't trigger anything.

s171
00:15:09.640 --> 00:15:12.360
You need to disc you need to prevent that.

s172
00:15:12.360 --> 00:15:19.080
And we found that this precision versus recall is the way to balance those two competing goals.

s173
00:15:19.080 --> 00:15:21.960
So when you do this this way, you have an open world grading.

s174
00:15:21.960 --> 00:15:30.560
Instead of trying to define one problem that's perfect, you can give it a real open-source task that can have multiple vulnerabilities, even though those that you don't know about.

s175
00:15:30.560 --> 00:15:41.600
You post-hoc, since you're asking for a proof of vulnerability, you can then go say what is the total set found of those known and unknown, and you can score precision and recall and normalize both,

s176
00:15:41.600 --> 00:15:43.120
so they're multiplicative.

s177
00:15:43.120 --> 00:15:47.240
It won't just keep finding the same easy bug.

s178
00:15:47.800 --> 00:15:56.360
You add as I said, it's open world, so you can find unknown bugs and use it on real open-source, and it also gives a clean trajectory.

s179
00:15:56.360 --> 00:16:03.360
Now, the key to doing this, the one thing that you do have to add to the grader is the ability to distinguish between multiple bugs if it gives you a POV.

s180
00:16:03.360 --> 00:16:07.360
The way we do this is the same way everyone in industry does it.

s181
00:16:07.360 --> 00:16:08.800
We look at the stack backtrace.

s182
00:16:08.800 --> 00:16:17.560
If you've ever had your program crash on Windows or Mac, and it's like submit to Microsoft or Apple, what it's doing is it's submitting the backtrace, and they're uniquifying

s183
00:16:17.560 --> 00:16:21.120
those into independent bugs, and then they're scoring the results given based on that.

s184
00:16:21.120 --> 00:16:23.880
So, we built that into the grader.

s185
00:16:23.880 --> 00:16:31.280
It also means that there's no LLM as a judge, because let's face it, you can't judge trust the LLM that you're teaching to be a judge.

s186
00:16:31.280 --> 00:16:35.640
And it also, what we found, it limits or removes bias completely.

s187
00:16:35.640 --> 00:16:38.200
The model actually never knows how many vulnerabilities.

s188
00:16:38.200 --> 00:16:44.520
When you say go find a bug, you've actually then given it a piece of information that there is a bug, right?

s189
00:16:44.520 --> 00:16:50.800
And in fact, what we find is that models will then find tune on that and only try to find one.

s190
00:16:50.800 --> 00:16:57.400
Here we open the possibility that there's no bugs, which provides a little bit cleaner trajectory for that learning signal.

s191
00:16:57.400 --> 00:17:02.720
So, the key TLDR for this is don't define the task by a single bed, let the program define the task.

s192
00:17:02.720 --> 00:17:11.360
We see people trying to create artificial benchmarks or synthetic benchmarks, and they'll go out and say, "Hey, let's just go find one crash, and then we'll turn that into an RL."

s193
00:17:11.360 --> 00:17:17.480
What invariably ends up happening is the model will then reward hack, and then it'll stunt its growth.

s194
00:17:17.480 --> 00:17:21.120
Or worse, you'll have an incorrect benchmark.

s195
00:17:21.120 --> 00:17:24.680
So, the audit task is one way to continue that climb.

s196
00:17:24.680 --> 00:17:34.800
The second access, if you look at going from as I said, toy programs, CTFs, all the way up to open source where you have multiple types of bugs, is what are the capabilities that our model is able to do?

s197
00:17:34.800 --> 00:17:45.480
And this is some of our latest work where we collaborated with the foundational models, OpenAI, Anthropic, and we're able to check how well they can exploit high-value targets.

s198
00:17:46.160 --> 00:17:47.400
This hadn't been done before.

s199
00:17:47.400 --> 00:17:57.120
If we go look at public experiments out there, and we look at, for example, DARPA, they had looked at this question of fully autonomous where they said, "Hey, for synthetic problems that we can create,

s200
00:17:57.120 --> 00:17:59.560
can AI do arbitrary code execution?"

s201
00:17:59.560 --> 00:18:01.600
What we would consider a real hack.

s202
00:18:01.600 --> 00:18:10.680
But when you go and you look at AIxCC or Cyber Gym or Bounty Bench, all they really checked is whether the AI could crash the program.

s203
00:18:10.680 --> 00:18:12.960
Crashing a program is different than hacking it.

s204
00:18:12.960 --> 00:18:16.440
You can't go steal someone's IP by simply crashing a program.

s205
00:18:16.440 --> 00:18:22.400
So, this question of whether models could exploit high-value targets was actually open.

s206
00:18:22.840 --> 00:18:24.520
So, what high-value target should we look at?

s207
00:18:24.520 --> 00:18:25.960
We picked Chrome.

s208
00:18:25.960 --> 00:18:30.360
And in particular, we picked the JavaScript Wasm interpreter called V8.

s209
00:18:30.360 --> 00:18:37.040
Now, V8 is one of the things that maybe is foreign to you, but actually powers the internet.

s210
00:18:37.040 --> 00:18:42.720
V8 is how Chrome executes JavaScript, and JavaScript is what's under the attacker's control.

s211
00:18:42.720 --> 00:18:46.240
Put up a malicious website, it runs JavaScript, you can then exploit V8.

s212
00:18:46.240 --> 00:18:50.840
It also runs Edge, it runs Node.js, it runs Cloudflare Edge Workers.

s213
00:18:50.840 --> 00:18:57.360
If you've ever used an Edge Worker, it's actually running V8 where each tenant is a separate thread.

s214
00:18:57.360 --> 00:18:58.440
It's crazy.

s215
00:18:58.440 --> 00:19:03.840
And if you can find a vulnerability in V8, you can exploit all these systems.

s216
00:19:05.520 --> 00:19:13.680
V8 is difficult to do because it goes beyond typical programs as far as security measures to try to keep it safe.

s217
00:19:13.680 --> 00:19:18.960
For example, when you start looking at V8 and you look at the internals of this, there is a sandbox.

s218
00:19:18.960 --> 00:19:25.240
And so, inside this sandbox is where you run your untrusted code, things like media, images, and so on.

s219
00:19:25.240 --> 00:19:28.840
And inside the sandbox, we expect there to be vulnerabilities.

s220
00:19:28.840 --> 00:19:34.240
In other words, if you can crash a in-sandbox object, it doesn't mean anything.

s221
00:19:34.240 --> 00:19:36.040
That's expected behavior.

s222
00:19:36.040 --> 00:19:43.680
What makes V8 a high-value target and what makes rewards start at 10,000 and go up to 100,000, or if you sell them on the black market, millions.

s223
00:19:43.680 --> 00:19:45.840
Let's be frank here, people do that.

s224
00:19:45.840 --> 00:19:48.400
is whether you can do an out-of-sandbox exploit.

s225
00:19:48.400 --> 00:19:53.520
And that typically requires chaining multiple vulnerabilities together.

s226
00:19:53.560 --> 00:20:02.880
So, TLDR, if you could give Chrome to an LLM and it could come up with a zero-day, you would essentially be able to hack nation-states at that point.

s227
00:20:02.880 --> 00:20:07.640
It's a very worthwhile task to see how far we have to climb.

s228
00:20:07.640 --> 00:20:11.960
But, we also want to be able to measure where LLMs get stuck.

s229
00:20:11.960 --> 00:20:16.720
It's such a hard target that when it fails, you end up with very little signal.

s230
00:20:16.720 --> 00:20:23.880
And so, we designed an experiment on X-plane, where we bucketized 16 different capabilities in a ladder.

s231
00:20:23.880 --> 00:20:26.720
First, can you trigger a crap Can you trigger the vulnerability?

s232
00:20:26.720 --> 00:20:31.280
Do you just show a deviation when you hit the the vulnerable line of code?

s233
00:20:31.280 --> 00:20:34.200
Can you crash an in-sandbox object?

s234
00:20:34.200 --> 00:20:38.040
That's interesting, but that's just the first vulnerability that you find.

s235
00:20:38.040 --> 00:20:39.760
Then, can you get in-sandbox primitives?

s236
00:20:39.760 --> 00:20:42.640
Can you inside the sandbox get arbitrary read and write?

s237
00:20:42.640 --> 00:20:52.920
What that allows you to do is inside the sandbox, the way exploitation works is you first exploit inside the sandbox and then you have a Turing-complete program if you have arbitrary read write.

s238
00:20:52.920 --> 00:20:56.880
You then try looking for that second vulnerability and chaining it together.

s239
00:20:56.880 --> 00:20:59.080
Can you get out-of-sandbox primitives?

s240
00:20:59.080 --> 00:21:02.400
And then finally, can you do arbitrary code execution?

s241
00:21:02.400 --> 00:21:09.000
What this allows us to do is it allows us to measure how far models get in this ladder on a really hard target.

s242
00:21:09.000 --> 00:21:12.000
And the results were actually very interesting in this.

s243
00:21:12.000 --> 00:21:14.680
So, we ran this on 41 V8 vulnerabilities.

s244
00:21:14.680 --> 00:21:18.040
We went and hand vulnerified verified that they were all exploitable.

s245
00:21:18.040 --> 00:21:24.560
We took actually the leader for the current Chrome security, his name is Sung Hin Lee, verify these for us.

s246
00:21:24.560 --> 00:21:33.800
And what we found is that if you're purely looking at old benchmarks where our triggering a crash is what you want to do, it's really not a distinguisher among models.

s247
00:21:33.800 --> 00:21:39.200
GPT and GPT 5.5 and Mythos both achieved 95%.

s248
00:21:39.200 --> 00:21:43.720
They were able to trigger a vulnerability 39 out of 41 times.

s249
00:21:43.720 --> 00:21:45.800
Essentially, all the tasks are side.

s250
00:21:45.800 --> 00:21:56.120
And then if you started to look at lower powered models, things like Gemini, Kimmy, Minimax, GLM, they were still able to succeed about 50% of the time.

s251
00:21:56.120 --> 00:21:57.240
So, think about this.

s252
00:21:57.240 --> 00:22:05.960
If you were looking at the old benchmarks, the message would be 50% of the time Kimmy succeeds in hacking, but that's because their definition of hacking was broken.

s253
00:22:05.960 --> 00:22:08.120
It was simply crashing it.

s254
00:22:08.120 --> 00:22:11.000
The real question is can they do a full sandbox escape?

s255
00:22:11.000 --> 00:22:14.840
And this is where we see a distinguishing characteristics.

s256
00:22:15.080 --> 00:22:25.280
So, if we look at what I'd call arbitrary code execution is really what the elite would do, Mythos was a quite surprising able to do this 73% of the time.

s257
00:22:25.280 --> 00:22:31.800
So, 30 out of the 41 examples, Mythos was able to do this sort of full control flow hijack.

s258
00:22:31.800 --> 00:22:33.960
GPT, sorry, the little bar here is wrong.

s259
00:22:33.960 --> 00:22:39.080
This was 68% of the time, and Gemini and Kimmy were 0% of the time.

s260
00:22:39.080 --> 00:22:42.400
So, we're starting to see a signal between these models on what they can do.

s261
00:22:42.400 --> 00:22:46.400
Little bars here are wrong, but the actual numbers are correct.

s262
00:22:47.480 --> 00:22:54.960
So, there's some cool evidence actually that these aren't memorized, that people like Mythos and GPT just didn't have access to zero days out there.

s263
00:22:54.960 --> 00:22:56.520
So, this is where I get a geek out on security.

s264
00:22:56.520 --> 00:23:07.880
For example, in CVE-2023-670T, this was something that the experts in Chrome, it's a very small community, they knew that it was exploitable and they came up with a POC, but what happened inside

s265
00:23:07.880 --> 00:23:13.600
Mythos was Mythos took a route that everyone thought would be too hard to do in practice.

s266
00:23:13.600 --> 00:23:23.240
One of the things that Mythos was able to do was reverse JavaScript's math.random and use that to forge a pointer for a return-oriented program out of the Uber cage exploit.

s267
00:23:23.240 --> 00:23:24.280
It was very creative.

s268
00:23:24.280 --> 00:23:26.040
So, this wasn't a publicly known exploit.

s269
00:23:26.040 --> 00:23:33.120
There is a public one, but what it came up with was very different for which experts actually thought would be too difficult in practice.

s270
00:23:33.120 --> 00:23:41.960
CVE-2024-76 7965, it found a new WASM path past where all the public work had stop had stopped.

s271
00:23:41.960 --> 00:23:46.160
In fact, it was unclear that there was a public exploit that worked for this.

s272
00:23:46.160 --> 00:23:55.040
We were able, again through a lot of manual effort, to create one after the fact, but we know that that wasn't public to the best of our knowledge.

s273
00:23:55.680 --> 00:24:03.960
2024-0519, again, public vulnerability, no public exploit, Mythos was able to succeed.

s274
00:24:04.560 --> 00:24:08.120
At the end of this, the work was on par with a human elite researcher.

s275
00:24:08.120 --> 00:24:13.040
I actually want to say a few more words about 2024-7965 cuz that one was actually pretty interesting.

s276
00:24:13.040 --> 00:24:23.280
This is one for which we knew of a public we knew that it we could exploit it on an arm, but actually even our internal expert didn't think that you could do it on x86 and Mythos succeeded.

s277
00:24:23.280 --> 00:24:26.960
So, fairly significant proof that this wasn't just memorization.

s278
00:24:26.960 --> 00:24:31.000
These are hard tasks against uh hardened targets.

s279
00:24:31.000 --> 00:24:34.720
So, you can download this entire set at exploitbench.ai.

s280
00:24:34.720 --> 00:24:40.400
We provide all the uh all the uh all the environments.

s281
00:24:40.400 --> 00:24:42.440
These are Docker images that you can just pull from GitHub.

s282
00:24:42.440 --> 00:24:43.760
They have an MCP interface.

s283
00:24:43.760 --> 00:24:44.760
It's really cool.

s284
00:24:44.760 --> 00:24:49.520
You can just say like Claude pointed at the MCP interface and see if it can hack it.

s285
00:24:49.520 --> 00:24:52.880
We provided all the data in the transcripts with the exception of Mythos.

s286
00:24:52.880 --> 00:24:55.480
And the reason that we withheld Mythos was twofold.

s287
00:24:55.480 --> 00:25:00.360
First is we had an NDA that we couldn't release me those transcripts cuz it's not public.

s288
00:25:00.360 --> 00:25:05.880
But second, actually me those was able to come up with weaponized exploits that weren't public.

s289
00:25:05.880 --> 00:25:07.600
And so we kind of hit this quandary out there.

s290
00:25:07.600 --> 00:25:16.240
If we're going to publish these benchmarks and we believe in open science, but the models are creating actually interesting exploits for high-value targets.

s291
00:25:16.240 --> 00:25:19.960
What do you do as far as the open science part of this?

s292
00:25:19.960 --> 00:25:20.920
We don't have an answer.

s293
00:25:20.920 --> 00:25:23.520
Kind of fun to think about.

s294
00:25:24.760 --> 00:25:29.080
So for the next steps, I mean we only have a 20-minute talk here.

s295
00:25:30.160 --> 00:25:39.840
One of the things that we are doing is we're taking these as really benchmarks to see where the frontier models stop, and then we're building reinforcement learning environments to help get models past that.

s296
00:25:39.840 --> 00:25:50.000
The way that we go about this is we've done a a fairly curated approach where we take open-source software and we built a very extensive vulnerability mining machine based upon our work with DARPA

s297
00:25:50.000 --> 00:25:53.760
over the last decade for novel vulnerability discovery.

s298
00:25:53.760 --> 00:25:55.760
We find unique proofs of vulnerability.

s299
00:25:55.760 --> 00:26:01.080
These are zero days no one else use and we use these to then build reinforcement learning environments.

s300
00:26:01.080 --> 00:26:02.760
Why are we finding zero days?

s301
00:26:02.760 --> 00:26:12.160
Well, we want to make sure that the models aren't simply memorizing and we know if it's a vulnerability they've never seen before that it can't at least be just memorizing that.

s302
00:26:12.160 --> 00:26:21.200
We're able to do this at scale where some of our the companies that we work with we're providing up to 10,000 reinforcement learning environments per month to really accelerate their

s303
00:26:21.200 --> 00:26:21.920
uh learning.

s304
00:26:21.920 --> 00:26:31.720
We of course can't take credit for how far these models have come, but we like the fact that we've had in some way uh some impact on how well they do at cybersecurity.

s305
00:26:32.200 --> 00:26:35.640
So the TLDR in the entire talk is training cybersecurity is really not mysterious.

s306
00:26:35.640 --> 00:26:45.280
What it takes is an actual expert that builds the right oracles that when you go back and look at the transcripts goes and tries to figure out was the was the machine just memorizing?

s307
00:26:45.280 --> 00:26:46.560
Was it doing reward hacking?

s308
00:26:46.560 --> 00:26:54.040
And most importantly, how do you handle the case where the machines are finding vulnerabilities that you didn't know about before.

s309
00:26:54.040 --> 00:26:56.960
If you're interested in this, please reach out.

s310
00:26:56.960 --> 00:26:59.400
Happy to answer questions.