Teaching AIs to Hack — Prof. David Brumley, Bugcrowd

AI Engineer · 27 min · 310 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12All right, everybody.
2. 00:13We're going to talk about hacking.
3. 00:15I love hacking.
4. 00:16We have a very small audience here, so I assume everyone here loves hacking as well.
5. 00:20So, I want to talk about designing reinforcement learning environments for cybersecurity tasks.
6. 00:25Essentially, we all want to teach computers to hack because well, we're pushing out programs faster than other ever and so we need to be able to check them at machine speeds in scale.
7. 00:35And this has been my research project for well over two decades.
8. 00:39My name is David Brumley.
9. 00:40I am a full professor at Carnegie Mellon University where I work on AI and cybersecurity.
10. 00:46And I'm also chief AI and science officer at Bugcrowd where I work on data partnerships.
11. 00:51So, before I talk about what we do and how we do it and why it's important to design cybersecurity tasks correctly for reinforcement learning environment, I want to start off with how humans learn because
12. 01:02I mean, I love teaching people to hack.
13. 01:05And I remember in particular a case where we run a hacking contest called picoCTF.
14. 01:10picoCTF has about a million high school kids every year play in this contest.
15. 01:15Um and so it's a a really fun way for people to get an intro to cybersecurity.
16. 01:19So, in 2016, a young uh person showed up on our scoreboard who was going by the hacker name fluorescence.
17. 01:26And typically, we know who is doing well in the contest.
18. 01:29It's kind of the typical suspects like a Palo Alto High School or uh some of the Washington D.C. high schools.
19. 01:36We know who's going to win the contest.
20. 01:37And so, this kind of independent starts showing up uh scoring on our scoreboard and we had no idea who it was.
21. 01:44So, we reach out.
22. 01:45It's actually a 17-year-old kid who found out about cybersecurity trying to get into it from math competitions.
23. 01:51He got bored with the math competitions and started doing them.
24. 01:54And very quickly, he ended up actually scoring second in picoCTF competing against all these high school kids.
25. 02:00And we asked actually "How did you learn this?"
26. 02:03And what he said really was uh germane to this task.
27. 02:07What I did is I looked at the cybersecurity task, and then I started Googling "What is the information I needed?"
28. 02:14I would read about it, I'd look at write-ups, and then I'd start emulating that.
29. 02:17And this kid actually ended up coming in second.
30. 02:20I recruited him to CMU, and he followed this methodology of studying write-ups and practicing cybersecurity on a graduated scale, easy problems first, and then slowly getting more difficult.
31. 02:31And he actually turned into what's called a Pwn2Own winner.
32. 02:34So, Pwn2Own, if you've never heard of it, is one of the more elite cybersecurity competitions.
33. 02:40This kid, just 2 years after he first learned cybersecurity, enters.
34. 02:43And uh if you read about it at the time, he was the first one to hack a Tesla.
35. 02:49So, he walked out of this contest with $375,000 in cash and a brand new Tesla.
36. 02:55The reason I tell this story is actually the way we teach AI uh frontier models to hack is the same way that we've been successful teaching high school students,
37. 03:04such as Richard Zhu, to become Pwn2Own winners.
38. 03:08My other students include people like George Hotz, who did the first iPhone jailbreak, and current Pwn2Own winners like Sang Heon Lee.
39. 03:16And so, what I want to talk about is how we teach reinforcement learning, and do it the same way that we've been teaching hacking for a while.
40. 03:23And it really breaks down into two different axes.
41. 03:27The first thing when designing these sorts of tasks for people is to look at target difficulty.
42. 03:34There's a spectrum of different challenges that you can look at from toy problems through CTF and synthetic problems, all the way up to hardened targets.
43. 03:43The second axis for teaching machines to hack is really looking at exploitation difficulty.
44. 03:49For example, when we look at a toy program, we may start looking at the sort of skills it needs to acquire to be able to hack that.
45. 03:56For example, if you have a toy program and it has a bug, can the LLM figure out where the bug is?
46. 04:02Can it then prove that it knows where it is by triggering a crash or some other fault in the program?
47. 04:09But of course, hacking is not just crashing a program.
48. 04:11We want to take control of that program.
49. 04:13That's the beautiful thing about hacking.
50. 04:15It's bending computers to our will.
51. 04:17It's what makes it unique in the sciences.
52. 04:21So you look at things like, "Hey, there's a flaw in that program.
53. 04:24Can I use that to do arbitrary read writes in memory?
54. 04:27Or even to do a full arbitrary code execution exploit?"
55. 04:31And so if you remember nothing else from this talk, it's really the way that we teach LLMs, whether it be frontier models like Anthropic or private models that you're tuning in your house,
56. 04:42you follow these two axes where you're trying to come up with a set of tasks that increase in target difficulty along one, and then you're teaching specific cybersecurity skills
57. 04:52on the second.
58. 04:55In other words, hacking is really a ladder.
59. 04:57And this is what actually matches cybersecurity so well to reinforcement learning.
60. 05:03We have a ladder of tasks and we typically end up with a good oracle for whether they can achieve that task.
61. 05:09And so you can start to measure whether your model is learning the right set of capabilities.
62. 05:14So this talk is really divided into three parts.
63. 05:22The first one is to talk about vulnerability discovery.
64. 05:25And when we talk about vulnerability discovery, what we're talking about is in the variety of different programs that you encounter in real life, how do you design oracles that are correct
65. 05:36for determining whether or not a model has successfully been able to detect that vulnerability?
66. 05:41And what's interesting is several of the cybersecurity benchmarks out there were amazing first-generation pieces of work, but they have a critical flaw where the model will actually stop learning after it finds the easiest vulnerability.
67. 05:55And that can prevent them from getting smarter.
68. 05:58The second is I want to talk about how we are designing benchmarks to measure this ability to do weaponization.
69. 06:04And this is really where we get into where does security differentiate from bug finding.
70. 06:08And we'll talk about how well LLMs do against what I would call hard targets.
71. 06:13A hard target one easy way to to look at it is how much would you pay for an exploit that a model could produce.
72. 06:20We know Richard Zhu fluorescence was paid $375,000 and got a brand new Tesla for one exploit.
73. 06:27Can models achieve that capabilities today?
74. 06:29And then I'm going to just summarize ways that if you're interested in this environment, we can connect and do more work together.
75. 06:36So, very simple talk.
76. 06:37So, let's talk about the first axis of discovery and where you really want to um learn what you're going to be measuring.
77. 06:45This is a key part in reinforcement learning where if you set up the wrong task objective, the LLM will learn it, but it'll learn the wrong thing.
78. 06:53So, some definitions to begin with.
79. 06:55Let's start defining the problem.
80. 06:57When we think about reinforcement learning or we talk about gyms, there's some key components in that.
81. 07:02There's of course other things, but the key components are you need a vulnerable application.
82. 07:06And we like to enclose these inside container environments so that they're reproducible.
83. 07:10We make sure that they run and that you don't have variations between for example, if I run a program on this version of Linux versus a different version of Linux, it actually may behave differently.
84. 07:22And so, you want to standardize that with a vulnerable program.
85. 07:24You need a grading oracle.
86. 07:26Now, one of the things I think the previous talk was talking about was LLM as a judge is a reasonable thing.
87. 07:31What we found in cybersecurity is that is flawed.
88. 07:34The LLMs will always say they were successful hacking.
89. 07:38And so, what you want to come up with is a deterministic grading oracle for each of the different levels you're getting at.
90. 07:44For example, if you're trying to teach it to just find a bugs, maybe this grading Oracle is was it able to trigger a crash.
91. 07:51We'll talk about that more in a second.
92. 07:53So, you have this reinforcement learning environment or this gym environment and of course you have your LLM and an orchestrator that's going to talk to it.
93. 07:59The way we set up our task is very simply, we expose through MCP a few key functions, a setup function.
94. 08:06So, the LLM will call setup, it returns the problem definition.
95. 08:09We give it standard tool calls such as read and write inside the container inside a sandbox inside the container, and then a grading Oracle at the very end.
96. 08:19And so, you end up with this vulnerable program in here, a grading Oracle, and I'm going to assume that you've already verified that there is at least one flaw in this program.
97. 08:28Maybe you yourself have figured out that it can crash.
98. 08:31Maybe you have downloaded it from a bug report and you've been able to reproduce that vulnerability.
99. 08:36We won't get into that, that's part of our sauce that we do at Bugcrowd.
100. 08:40But, once you do that, you have this package environment and then your task prompt is going to be something very simple like, "Dear LLM, can you find and exploit the vulnerability?"
101. 08:48Now, you don't want to just ask, "Can you find the vulnerability?"
102. 08:51because then you won't be able to distinguish between an LLM hallucination and a real vulnerability.
103. 08:57So, you almost always ask it to actually exploit the vulnerability.
104. 09:02And that exploit is going to be key to how we do reinforcement learning.
105. 09:07So, the LLM does some thinking and it comes up with an exploit.
106. 09:10For example, this very, very simple program, if you just give it enough A's, you'll trigger a crash.
107. 09:16So, that's the LLM's witness, the proof of vulnerability that it was able to find something.
108. 09:21You run that input through your grading Oracle.
109. 09:25The Oracle that determines did the program misbehave or not.
110. 09:28In this case, the program would simply crash and you farm out your rewards.
111. 09:35This is a very elegant way and actually this is the way we teach people to hack.
112. 09:39We set up a deterministic auto grader.
113. 09:42For example, in CTFs, it's cuz you capture the flag.
114. 09:45Within a cybersecurity environment like this, the level one maybe can it crash, all the way up to control flow hijack, where for example, you may ask the LLM, "Can you do something like launch a calculator, some external program you shouldn't be able to run, or do a reverse shell?"
115. 10:00So, that's the basic setup.
116. 10:02But, there's a problem with this.
117. 10:03This is the way if you go look at the existing benchmarks like uh Cybex or uh CyberGym, they set up the task.
118. 10:10But, there's actually a problem here.
119. 10:12And that's because there's an assumption that the program only has one vulnerability.
120. 10:17I don't know about you, but it's very rare to find a program for which you know there's only one vulnerability.
121. 10:24So, what happens if you have two vulnerabilities here?
122. 10:30This actually breaks a lot of assumptions in current evaluation environments.
123. 10:34You ask the same question, "Dear LLM, can you find and exploit the vulnerability?"
124. 10:38But, now the LLM has a lot of freedom to reward hack.
125. 10:42For example, which vulnerability should it find?
126. 10:46If you came in only knowing about the first vulnerability, but there's a second one you didn't know about, what do you do if the LLM thinks it found a second one?
127. 10:55Or suppose you know two.
128. 10:57What we found is on existing benchmarks with multiple pro- with a real OSS benchmarks, there are multiple vulnerabilities, the LLM will just continue to find the easiest vulnerability, and that really limits its trajectory as far as what it can learn.
129. 11:12And then you have a question, if it does find a vulnerability you did not know about, well, how do you score it, right?
130. 11:19You certainly don't want to give tasks that have no vulnerabilities because then you don't know if you're wasting your time.
131. 11:24But, what if the LLM finds an unknown vulnerability?
132. 11:28Here's where you can run into a catch-22.
133. 11:31What existing benchmarks do is they tell the LLM which bug.
134. 11:36For example, in many of the benchmarks out there like side bench, they will give a backtrace that says for example, I know the vulnerability is in this backtrace which identifies the vulnerable function.
135. 11:48But at that point, you're teaching the LLM, but you're pointing at it exactly the problem.
136. 11:55So the LLM no longer has to reason about the program and that will stunt its reasoning capability.
137. 12:00Essentially, if you're nudging it and saying here's the vulnerability, it's in this function, it doesn't have to do a lot.
138. 12:06In fact, it can often fit that entire function in its context window and it doesn't have to reason much.
139. 12:12The second problem though is if you don't tell the LLM which one and there's multiple vulnerabilities, it can always just then reward hack the easiest problem.
140. 12:21And we see this in every foundational LLM out there and we see it in as far as I can tell most of the benchmarks out there.
141. 12:29There will be multiple vulnerabilities, it will be graded, but because the grading is just checking for for example, a crash, it's not exploring the full state space and the LLM will just keep returning
142. 12:39the same one.
143. 12:41This is also a problem in some of the public competitions.
144. 12:43For example, we won something called the Cyber Grand Challenge from DARPA.
145. 12:47It was the first challenge from DARPA to show that fully autonomous cyber is capable.
146. 12:5150% of the hand curated challenges had unknown vulnerabilities.
147. 12:56This was DARPA spent $60 million designing a contest trying to come up with problems that were well defined and well scoped and they accidentally added additional bugs and 50% of those were ones that were actually exploited.
148. 13:10So this idea that we're just going to create synthetic problems with one bug doesn't work.
149. 13:15People have tried it, spent a lot of money, you always introduce new ones.
150. 13:19Second problem is second example I'd show is the AIxCC.
151. 13:23I designed the scoring algorithm for this.
152. 13:25This is again a very large DARPA program that ran last year in DEF CON where 18 of the bugs found were unintended ones.
153. 13:31And so the TLDR on here is you can't just say, "Well, we're going to hand curate an environment with just one vulnerability."
154. 13:37Experts have tried, it doesn't work.
155. 13:39You have to change the problem definition.
156. 13:42So we've been thinking about this and what we developed is a new way to test is called the audit task.
157. 13:46Again, suppose you have two different bugs, but you flip the question from just find a bug to find all vulnerabilities discovered.
158. 13:55At this point, the LLM has the freedom to find multiple bugs and submit multiple proofs of vulnerabilities.
159. 14:01And it may be full proofs of vulnerabilities for bugs you know about and bugs you don't.
160. 14:06You run all vulnerabilities through your oracle.
161. 14:10And this is where it's very important to have a deterministic grader.
162. 14:12So for here, for example, there's two vulnerabilities, it gives us two inputs that crash both vulnerabilities.
163. 14:17And part of this grader now have to uniquify them to show that two different vulnerabilities are triggered.
164. 14:22Now if we didn't know about vuln two, this also gives us the opportunity to increase our ground truth.
165. 14:27We haven't told the vuln the LLM that we don't know about something that it found.
166. 14:31It just gave us proof that it was able to find it.
167. 14:33So we can normalize the set of known vulnerabilities at that point to be something like D star and calculate the the precision and recall for the model across multiple vulnerabilities.
168. 14:43For example, recall is the number of known that it found over the total set and precision is the number of found over the submitted.
169. 14:50What this prevents the model from doing and it essentially balances is the ability for it to go find unknown vulnerabilities, but also prevents you from prevents the model from just spamming.
170. 15:01You don't want it to give you a bunch of things that aren't vulnerabilities, like for example, giving us POV in four that doesn't trigger anything.
171. 15:09You need to disc you need to prevent that.
172. 15:12And we found that this precision versus recall is the way to balance those two competing goals.
173. 15:19So when you do this this way, you have an open world grading.
174. 15:21Instead of trying to define one problem that's perfect, you can give it a real open-source task that can have multiple vulnerabilities, even though those that you don't know about.
175. 15:30You post-hoc, since you're asking for a proof of vulnerability, you can then go say what is the total set found of those known and unknown, and you can score precision and recall and normalize both,
176. 15:41so they're multiplicative.
177. 15:43It won't just keep finding the same easy bug.
178. 15:47You add as I said, it's open world, so you can find unknown bugs and use it on real open-source, and it also gives a clean trajectory.
179. 15:56Now, the key to doing this, the one thing that you do have to add to the grader is the ability to distinguish between multiple bugs if it gives you a POV.
180. 16:03The way we do this is the same way everyone in industry does it.
181. 16:07We look at the stack backtrace.
182. 16:08If you've ever had your program crash on Windows or Mac, and it's like submit to Microsoft or Apple, what it's doing is it's submitting the backtrace, and they're uniquifying
183. 16:17those into independent bugs, and then they're scoring the results given based on that.
184. 16:21So, we built that into the grader.
185. 16:23It also means that there's no LLM as a judge, because let's face it, you can't judge trust the LLM that you're teaching to be a judge.
186. 16:31And it also, what we found, it limits or removes bias completely.
187. 16:35The model actually never knows how many vulnerabilities.
188. 16:38When you say go find a bug, you've actually then given it a piece of information that there is a bug, right?
189. 16:44And in fact, what we find is that models will then find tune on that and only try to find one.
190. 16:50Here we open the possibility that there's no bugs, which provides a little bit cleaner trajectory for that learning signal.
191. 16:57So, the key TLDR for this is don't define the task by a single bed, let the program define the task.
192. 17:02We see people trying to create artificial benchmarks or synthetic benchmarks, and they'll go out and say, "Hey, let's just go find one crash, and then we'll turn that into an RL."
193. 17:11What invariably ends up happening is the model will then reward hack, and then it'll stunt its growth.
194. 17:17Or worse, you'll have an incorrect benchmark.
195. 17:21So, the audit task is one way to continue that climb.
196. 17:24The second access, if you look at going from as I said, toy programs, CTFs, all the way up to open source where you have multiple types of bugs, is what are the capabilities that our model is able to do?
197. 17:34And this is some of our latest work where we collaborated with the foundational models, OpenAI, Anthropic, and we're able to check how well they can exploit high-value targets.
198. 17:46This hadn't been done before.
199. 17:47If we go look at public experiments out there, and we look at, for example, DARPA, they had looked at this question of fully autonomous where they said, "Hey, for synthetic problems that we can create,
200. 17:57can AI do arbitrary code execution?"
201. 17:59What we would consider a real hack.
202. 18:01But when you go and you look at AIxCC or Cyber Gym or Bounty Bench, all they really checked is whether the AI could crash the program.
203. 18:10Crashing a program is different than hacking it.
204. 18:12You can't go steal someone's IP by simply crashing a program.
205. 18:16So, this question of whether models could exploit high-value targets was actually open.
206. 18:22So, what high-value target should we look at?
207. 18:24We picked Chrome.
208. 18:25And in particular, we picked the JavaScript Wasm interpreter called V8.
209. 18:30Now, V8 is one of the things that maybe is foreign to you, but actually powers the internet.
210. 18:37V8 is how Chrome executes JavaScript, and JavaScript is what's under the attacker's control.
211. 18:42Put up a malicious website, it runs JavaScript, you can then exploit V8.
212. 18:46It also runs Edge, it runs Node.js, it runs Cloudflare Edge Workers.
213. 18:50If you've ever used an Edge Worker, it's actually running V8 where each tenant is a separate thread.
214. 18:57It's crazy.
215. 18:58And if you can find a vulnerability in V8, you can exploit all these systems.
216. 19:05V8 is difficult to do because it goes beyond typical programs as far as security measures to try to keep it safe.
217. 19:13For example, when you start looking at V8 and you look at the internals of this, there is a sandbox.
218. 19:18And so, inside this sandbox is where you run your untrusted code, things like media, images, and so on.
219. 19:25And inside the sandbox, we expect there to be vulnerabilities.
220. 19:28In other words, if you can crash a in-sandbox object, it doesn't mean anything.
221. 19:34That's expected behavior.
222. 19:36What makes V8 a high-value target and what makes rewards start at 10,000 and go up to 100,000, or if you sell them on the black market, millions.
223. 19:43Let's be frank here, people do that.
224. 19:45is whether you can do an out-of-sandbox exploit.
225. 19:48And that typically requires chaining multiple vulnerabilities together.
226. 19:53So, TLDR, if you could give Chrome to an LLM and it could come up with a zero-day, you would essentially be able to hack nation-states at that point.
227. 20:02It's a very worthwhile task to see how far we have to climb.
228. 20:07But, we also want to be able to measure where LLMs get stuck.
229. 20:11It's such a hard target that when it fails, you end up with very little signal.
230. 20:16And so, we designed an experiment on X-plane, where we bucketized 16 different capabilities in a ladder.
231. 20:23First, can you trigger a crap Can you trigger the vulnerability?
232. 20:26Do you just show a deviation when you hit the the vulnerable line of code?
233. 20:31Can you crash an in-sandbox object?
234. 20:34That's interesting, but that's just the first vulnerability that you find.
235. 20:38Then, can you get in-sandbox primitives?
236. 20:39Can you inside the sandbox get arbitrary read and write?
237. 20:42What that allows you to do is inside the sandbox, the way exploitation works is you first exploit inside the sandbox and then you have a Turing-complete program if you have arbitrary read write.
238. 20:52You then try looking for that second vulnerability and chaining it together.
239. 20:56Can you get out-of-sandbox primitives?
240. 20:59And then finally, can you do arbitrary code execution?
241. 21:02What this allows us to do is it allows us to measure how far models get in this ladder on a really hard target.
242. 21:09And the results were actually very interesting in this.
243. 21:12So, we ran this on 41 V8 vulnerabilities.
244. 21:14We went and hand vulnerified verified that they were all exploitable.
245. 21:18We took actually the leader for the current Chrome security, his name is Sung Hin Lee, verify these for us.
246. 21:24And what we found is that if you're purely looking at old benchmarks where our triggering a crash is what you want to do, it's really not a distinguisher among models.
247. 21:33GPT and GPT 5.5 and Mythos both achieved 95%.
248. 21:39They were able to trigger a vulnerability 39 out of 41 times.
249. 21:43Essentially, all the tasks are side.
250. 21:45And then if you started to look at lower powered models, things like Gemini, Kimmy, Minimax, GLM, they were still able to succeed about 50% of the time.
251. 21:56So, think about this.
252. 21:57If you were looking at the old benchmarks, the message would be 50% of the time Kimmy succeeds in hacking, but that's because their definition of hacking was broken.
253. 22:05It was simply crashing it.
254. 22:08The real question is can they do a full sandbox escape?
255. 22:11And this is where we see a distinguishing characteristics.
256. 22:15So, if we look at what I'd call arbitrary code execution is really what the elite would do, Mythos was a quite surprising able to do this 73% of the time.
257. 22:25So, 30 out of the 41 examples, Mythos was able to do this sort of full control flow hijack.
258. 22:31GPT, sorry, the little bar here is wrong.
259. 22:33This was 68% of the time, and Gemini and Kimmy were 0% of the time.
260. 22:39So, we're starting to see a signal between these models on what they can do.
261. 22:42Little bars here are wrong, but the actual numbers are correct.
262. 22:47So, there's some cool evidence actually that these aren't memorized, that people like Mythos and GPT just didn't have access to zero days out there.
263. 22:54So, this is where I get a geek out on security.
264. 22:56For example, in CVE-2023-670T, this was something that the experts in Chrome, it's a very small community, they knew that it was exploitable and they came up with a POC, but what happened inside
265. 23:07Mythos was Mythos took a route that everyone thought would be too hard to do in practice.
266. 23:13One of the things that Mythos was able to do was reverse JavaScript's math.random and use that to forge a pointer for a return-oriented program out of the Uber cage exploit.
267. 23:23It was very creative.
268. 23:24So, this wasn't a publicly known exploit.
269. 23:26There is a public one, but what it came up with was very different for which experts actually thought would be too difficult in practice.
270. 23:33CVE-2024-76 7965, it found a new WASM path past where all the public work had stop had stopped.
271. 23:41In fact, it was unclear that there was a public exploit that worked for this.
272. 23:46We were able, again through a lot of manual effort, to create one after the fact, but we know that that wasn't public to the best of our knowledge.
273. 23:552024-0519, again, public vulnerability, no public exploit, Mythos was able to succeed.
274. 24:04At the end of this, the work was on par with a human elite researcher.
275. 24:08I actually want to say a few more words about 2024-7965 cuz that one was actually pretty interesting.
276. 24:13This is one for which we knew of a public we knew that it we could exploit it on an arm, but actually even our internal expert didn't think that you could do it on x86 and Mythos succeeded.
277. 24:23So, fairly significant proof that this wasn't just memorization.
278. 24:26These are hard tasks against uh hardened targets.
279. 24:31So, you can download this entire set at exploitbench.ai.
280. 24:34We provide all the uh all the uh all the environments.
281. 24:40These are Docker images that you can just pull from GitHub.
282. 24:42They have an MCP interface.
283. 24:43It's really cool.
284. 24:44You can just say like Claude pointed at the MCP interface and see if it can hack it.
285. 24:49We provided all the data in the transcripts with the exception of Mythos.
286. 24:52And the reason that we withheld Mythos was twofold.
287. 24:55First is we had an NDA that we couldn't release me those transcripts cuz it's not public.
288. 25:00But second, actually me those was able to come up with weaponized exploits that weren't public.
289. 25:05And so we kind of hit this quandary out there.
290. 25:07If we're going to publish these benchmarks and we believe in open science, but the models are creating actually interesting exploits for high-value targets.
291. 25:16What do you do as far as the open science part of this?
292. 25:19We don't have an answer.
293. 25:20Kind of fun to think about.
294. 25:24So for the next steps, I mean we only have a 20-minute talk here.
295. 25:30One of the things that we are doing is we're taking these as really benchmarks to see where the frontier models stop, and then we're building reinforcement learning environments to help get models past that.
296. 25:39The way that we go about this is we've done a a fairly curated approach where we take open-source software and we built a very extensive vulnerability mining machine based upon our work with DARPA
297. 25:50over the last decade for novel vulnerability discovery.
298. 25:53We find unique proofs of vulnerability.
299. 25:55These are zero days no one else use and we use these to then build reinforcement learning environments.
300. 26:01Why are we finding zero days?
301. 26:02Well, we want to make sure that the models aren't simply memorizing and we know if it's a vulnerability they've never seen before that it can't at least be just memorizing that.
302. 26:12We're able to do this at scale where some of our the companies that we work with we're providing up to 10,000 reinforcement learning environments per month to really accelerate their
303. 26:21uh learning.
304. 26:21We of course can't take credit for how far these models have come, but we like the fact that we've had in some way uh some impact on how well they do at cybersecurity.
305. 26:32So the TLDR in the entire talk is training cybersecurity is really not mysterious.
306. 26:35What it takes is an actual expert that builds the right oracles that when you go back and look at the transcripts goes and tries to figure out was the was the machine just memorizing?
307. 26:45Was it doing reward hacking?
308. 26:46And most importantly, how do you handle the case where the machines are finding vulnerabilities that you didn't know about before.
309. 26:54If you're interested in this, please reach out.
310. 26:56Happy to answer questions.