WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7JgIS42mz7U

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7JgIS42mz7U.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.160 --> 00:00:06.640
Hey there, I'm Jack Cable and today I'm going to be talking about the effects of the AI bug apocalypse.

s2
00:00:06.640 --> 00:00:14.680
As you may have seen, frontier models are getting better than ever before at discovering and exploiting vulnerabilities in our software.

s3
00:00:14.680 --> 00:00:26.920
Right, this is leading to what many are calling a bug apocalypse where we're finding more and more vulnerabilities, particularly in the open-source libraries that power um all of the software we rely upon.

s4
00:00:26.920 --> 00:00:35.920
Right, so today I want to break down what exactly is happening and how defenders can get ahead of the exploitation that is occurring.

s5
00:00:35.920 --> 00:00:45.280
As far as my background, right now I'm the co-founder and CEO at Corridor, a company I started about 18 months ago focused on securing AI coding.

s6
00:00:45.280 --> 00:00:59.520
Before this, I served as a senior technical advisor in government at CISA, the Cybersecurity and Infrastructure Security Agency, where I worked with top software companies to help them build their products to be more secure by design.

s7
00:00:59.520 --> 00:01:01.200
I'm also an ethical hacker.

s8
00:01:01.200 --> 00:01:05.760
I got into the top 100 rank of hackers on when I was in high school.

s9
00:01:05.760 --> 00:01:07.680
I studied computer science at Stanford.

s10
00:01:07.680 --> 00:01:20.200
So I've seen firsthand how these simple, you know, repeat classes of vulnerabilities can be introduced and exploited and have been um close participant right in many of the the most recent

s11
00:01:20.200 --> 00:01:28.480
um advancements and seeing just what this means uh for both our adversaries as well as defenders.

s12
00:01:28.480 --> 00:01:36.600
Just to set the stage, right, as everyone here knows, I imagine AI coding tools are scaling faster than any software category in history.

s13
00:01:36.600 --> 00:01:48.520
Cursor, Copilot Code um grow exponentially and with that, right, also comes these improvements in how frontier models can find and exploit vulnerabilities.

s14
00:01:48.520 --> 00:01:51.560
Um so we're seeing, right, both ends of the equation shifting.

s15
00:01:51.560 --> 00:01:55.720
On one hand, um models can do better job finding vulnerabilities.

s16
00:01:55.720 --> 00:02:03.400
On the other hand, our attack surfaces are growing immensely, right, as AI becomes the default code writer.

s17
00:02:03.400 --> 00:02:06.440
Um so, what I want to explore in this talk, right, is how do we balance that?

s18
00:02:06.440 --> 00:02:12.760
How do we make sure that we're not going to have immensely more vulnerabilities than we've ever had before, right?

s19
00:02:12.760 --> 00:02:25.320
And just to to give some sense, right, I'll move myself here of um some of the statistics, right, um pulled some from last year where about 84% of developers were using AI coding tools, 30 to 40%

s20
00:02:25.320 --> 00:02:28.880
of companies encouraging use of AI coding assistants.

s21
00:02:28.880 --> 00:02:30.920
That was from Stack Overflow, right?

s22
00:02:30.920 --> 00:02:39.800
I don't I haven't seen the latest numbers this year, but what I would expect once those come out, right, is that is the vast, vast majority of developers and companies

s23
00:02:39.800 --> 00:02:41.959
who are using coding agents, right?

s24
00:02:41.959 --> 00:02:48.240
And part of this is the increasing level of autonomy by which these coding agents are being used.

s25
00:02:48.240 --> 00:02:54.360
It's no longer, you know, auto complete, often it's not even a developer synchronously within cursor.

s26
00:02:54.360 --> 00:03:06.080
Um right, when we do our own development, right, now it's um spinning up agents from within Slack or wherever folks are working and having many agents run at once in the background.

s27
00:03:06.080 --> 00:03:15.800
Um this is a tremendous shift in how software is being built, um and at the same time, right, like I mentioned, the frontier models are getting significantly better.

s28
00:03:15.800 --> 00:03:26.080
Um and you can look at it from, you know, pretty much any part of the cyber attack chain, ranging from finding vulnerabilities where models can, you know, now do do,

s29
00:03:26.080 --> 00:03:32.640
you know, significantly better than even I could, and you know, I've I've reported hundreds of vulnerabilities to various companies.

s30
00:03:32.640 --> 00:03:36.360
Um so, everything from finding vulnerabilities to exploiting them.

s31
00:03:36.360 --> 00:03:45.920
Uh this is a chart here that comes from Anthropic, right, showing Mythos compared to a number of um other models that they and others have put out.

s32
00:03:45.920 --> 00:03:58.600
Um and we can see that we're we're seeing quite rapid advancements in models' capabilities, right, And and particularly to execute more kind of autonomous attack chains.

s33
00:03:58.600 --> 00:04:02.200
So, as we think about adversaries who are using these models, right?

s34
00:04:02.200 --> 00:04:09.360
They're not just going to be discovering vulnerabilities, but they're going to be automating every part of the attack process.

s35
00:04:09.360 --> 00:04:10.560
So, it's our job, right?

s36
00:04:10.560 --> 00:04:19.000
As defenders to understand, okay, what are the points where we can make software systems more resilient to all of these attacks.

s37
00:04:19.000 --> 00:04:19.239
Right?

s38
00:04:19.239 --> 00:04:26.880
And to me, this brings back a lot of the work that I was doing in government around the the secure by design initiative, right?

s39
00:04:26.880 --> 00:04:28.800
Um and so so the overall question, right?

s40
00:04:28.800 --> 00:04:37.320
That I'm worried about is how can we make sure that frontier AI models aren't introducing exponentially more vulnerabilities over time, right?

s41
00:04:37.320 --> 00:04:49.240
Um even pre-AI, we've had this you know, uh heavy increase in common, relatively simple classes of vulnerabilities that are being exploited by adversaries.

s42
00:04:49.240 --> 00:04:51.960
AI is making this significantly easier, right?

s43
00:04:51.960 --> 00:04:58.040
So, I think the only way that we're going to to win as defenders is if we use the the same techniques, right?

s44
00:04:58.040 --> 00:04:59.960
To harden our systems.

s45
00:04:59.960 --> 00:05:01.919
Um and I would say that there is good news here, right?

s46
00:05:01.919 --> 00:05:10.760
That a lot of the vulnerabilities, pretty much all of the vulnerabilities that even frontier AI models are finding aren't anything new.

s47
00:05:10.760 --> 00:05:21.000
Yes, it's new that uh given vulnerability was found in a you know, specific file with with a piece of software, but that vulnerability class um isn't necessarily novel.

s48
00:05:21.000 --> 00:05:25.120
Um and and we can actually use that to our advantage, and I'll I'll get into that.

s49
00:05:25.120 --> 00:05:26.040
Um right?

s50
00:05:26.040 --> 00:05:31.240
So so overall, um the the thesis here is that um right?

s51
00:05:31.240 --> 00:05:40.200
We are seeing both attackers get more tools in their toolkit, and the same time, the way in which software is being built is fundamentally changing.

s52
00:05:40.200 --> 00:05:47.840
Um so so really the question then becomes how can we apply AI to shore up on software systems?

s53
00:05:47.840 --> 00:05:57.680
Um right, and I want to to take a quick detour um to some of the secure by design work that that I I kicked off with with others in government.

s54
00:05:57.680 --> 00:06:02.200
Um right, this is a paper that we put out in March of 2023.

s55
00:06:02.200 --> 00:06:13.680
So, just as you know, LLMs were starting to become more readily available, but um right, their application in coding at that time wasn't much more than, you know, auto complete.

s56
00:06:13.680 --> 00:06:18.800
And while that's useful, it wasn't necessarily the step change that we have now.

s57
00:06:18.800 --> 00:06:28.680
Um and what we focused on kind of laying out with this vision, right, was this idea that it it isn't really rocket science when it comes to preventing vulnerabilities

s58
00:06:28.680 --> 00:06:29.720
in software, right?

s59
00:06:29.720 --> 00:06:40.200
While it's true that it's hard to build a perfectly secure system, we do know how to build systems that are fundamentally more resilient to common classes of vulnerabilities.

s60
00:06:40.200 --> 00:06:48.400
Um and and just to, you know, make this concrete, um this is a set of vulnerability classes coming from MITRE.

s61
00:06:48.400 --> 00:06:56.600
It's the uh top classes that are um exploited in CISA's known exploited vulnerabilities catalog, right?

s62
00:06:56.600 --> 00:07:09.960
And if you go down this list, you'll you might notice, right, that pretty much all of these are basic types of vulnerabilities that not only have we known about for decades, but we've known how to prevent at scale for decades,

s63
00:07:09.960 --> 00:07:10.440
right?

s64
00:07:10.440 --> 00:07:13.520
Uh take buffer overflows, right, number two on that list.

s65
00:07:13.520 --> 00:07:19.080
Um and by the way, these are the same vulnerabilities that models like Mythos are finding in software.

s66
00:07:19.080 --> 00:07:27.680
Um and buffer overflows were first documented um about um I believe 30-plus years ago, right?

s67
00:07:27.680 --> 00:07:37.280
Um so we've had uh documented instances of how to find and exploit these vulnerabilities, and we also now have languages that are memory safe, right?

s68
00:07:37.280 --> 00:07:50.040
Languages like Rust, Go, pretty much any language um that that's not C or C++ um is built in a way such that it's impossible to introduce memory safety vulnerabilities, right?

s69
00:07:50.040 --> 00:07:54.560
They have guarantees that prevent those from being introduced.

s70
00:07:54.560 --> 00:08:03.120
So we have techniques by which we can prevent them and yet they continue getting introduced over and over again, right?

s71
00:08:03.120 --> 00:08:06.480
So so let's look at memory safety for instance.

s72
00:08:06.480 --> 00:08:20.160
There's you know the statistics range but approximately 60 to 70% of vulnerabilities in products written in memory unsafe languages can be completely prevented using memory safe languages, right?

s73
00:08:20.160 --> 00:08:31.520
And this is you know based on CV data out there and not only that right we've seen a lot of companies Google, Microsoft, Amazon even you know open source software

s74
00:08:31.520 --> 00:08:36.000
the Linux kernel is being rewritten in parts in rust.

s75
00:08:36.000 --> 00:08:43.919
We've seen real evidence that by shifting to memory safe languages you can reduce overall vulnerabilities, right?

s76
00:08:43.919 --> 00:08:52.560
On the right here is a chart from Google showing the rate of memory safety vulnerabilities over time in the Android operating system.

s77
00:08:52.560 --> 00:09:00.000
What's interesting right is that this isn't even you know they're they're not even necessarily rewriting code in a memory safe language.

s78
00:09:00.000 --> 00:09:11.600
They're just writing new code in a memory safe language and even then right the percent of memory safety vulnerabilities has dropped quite dramatically from you know about 75% in 2019

s79
00:09:11.600 --> 00:09:15.160
to maybe 30% in 2022.

s80
00:09:15.160 --> 00:09:17.320
So to me that's personally quite exciting, right?

s81
00:09:17.320 --> 00:09:24.120
Because it means that's not a given that we're going to continue having these basic vulnerabilities over and over again, right?

s82
00:09:24.120 --> 00:09:35.480
And in part of the you know high-level policy conversation I think as a result has to be not just how can we deploy these frontier models to find one-off vulnerabilities in software

s83
00:09:35.480 --> 00:09:43.600
that is something that we should be doing but at the same time, right, I don't want to miss out on opportunities to make our software fundamentally more secure.

s84
00:09:43.600 --> 00:10:00.360
Right, we could pour millions of dollars into essentially playing whack-a-mole with vulnerabilities and patching them one-off in some of the open-source libraries that we all rely on, or we could do a one-time rewrite, for instance, to move some of these critical libraries

s85
00:10:00.360 --> 00:10:03.680
into a language like Rust, right?

s86
00:10:03.680 --> 00:10:07.600
And then that will pay dividends for years to come.

s87
00:10:07.600 --> 00:10:10.800
So this is really at the core of how I'm thinking about this, right?

s88
00:10:10.800 --> 00:10:22.680
Is what are some of the fundamental changes that companies, that open-source developers can be making that can reduce exploitation both by models today, right, and models to come.

s89
00:10:22.680 --> 00:10:33.760
Cuz the advantage, right, of doing a rewrite, for instance, is that if you have some of these fundamental guarantees, then even if the models get smarter, right, Rust has programmatic

s90
00:10:33.760 --> 00:10:42.760
guarantees such that we know that memory-safe D vulnerabilities in no circumstances won't be possible to be introduced or discovered.

s91
00:10:42.760 --> 00:10:53.680
Right, and all of this comes in the context, too, of the fact that AI is increasingly capable at, of course, both writing code and then introducing vulnerabilities.

s92
00:10:53.680 --> 00:11:06.000
You might have seen a couple months ago one example where Opus 46, right, by all accounts a very smart model, introduced a vulnerability in a smart contract that led to a couple million dollars being stolen,

s93
00:11:06.000 --> 00:11:07.120
right?

s94
00:11:07.120 --> 00:11:13.600
So so while the models are very smart and capable, often times security is very contextual.

s95
00:11:13.600 --> 00:11:19.360
And the model just might not have the context in order to know that it's introducing a vulnerability, right?

s96
00:11:19.360 --> 00:11:23.400
And this is reflected in academic benchmarks.

s97
00:11:23.400 --> 00:11:39.520
One, for instance, here, Backsbench, you can find that backsbench.com by researchers at ETH Zurich, UC Berkeley, uh finds that even the best models introduce vulnerabilities about 20 to 40% of the time when writing code.

s98
00:11:39.520 --> 00:11:40.120
Right?

s99
00:11:40.120 --> 00:11:42.520
Um and this shouldn't necessarily come as a surprise.

s100
00:11:42.520 --> 00:11:43.760
Um for one, right?

s101
00:11:43.760 --> 00:11:51.840
Models are trained on all of the world's existing code, and humans haven't been great at not introducing vulnerabilities in code in the past.

s102
00:11:51.840 --> 00:11:52.960
But two, right?

s103
00:11:52.960 --> 00:12:02.440
Increasingly, and this kind of lines up with some of what we're seeing among our customers, is that the vulnerabilities being introduced are often and less so the basic one-liner

s104
00:12:02.440 --> 00:12:05.840
vulnerabilities, and more so contextual issues.

s105
00:12:05.840 --> 00:12:06.040
Right?

s106
00:12:06.040 --> 00:12:12.400
Things like authorization bugs that require an in-depth understanding of a company's business logic.

s107
00:12:12.400 --> 00:12:13.640
Um and that's something, right?

s108
00:12:13.640 --> 00:12:24.040
Even if the model's very smart, it's not being trained on your company's proprietary information or how your own um kind of, you know, threat model works.

s109
00:12:24.040 --> 00:12:36.680
Um and that is why, you know, I believe we're still seeing quite a high rate of vulnerability introduction um even by um, you know, by by all accounts very intelligent models.

s110
00:12:36.800 --> 00:12:47.040
Um so so let's now think about, okay, given that the vast majority of software development is being done with AI, how can we make sure that AI is capable of writing secure-by-design

s111
00:12:47.040 --> 00:12:47.960
software?

s112
00:12:47.960 --> 00:12:48.200
Right?

s113
00:12:48.200 --> 00:12:58.440
And part of this is a shift we're seeing in the level of autonomy um that AI is um now given when it comes to software development tasks.

s114
00:12:58.440 --> 00:12:58.640
Right?

s115
00:12:58.640 --> 00:13:13.280
We're kind of moving up this ladder that started with auto-complete to you know, agents within Cursor, Claude Code that can synchronously produce code, to now increasingly these autonomous agents that can work for an hour, hours at a time,

s116
00:13:13.280 --> 00:13:15.800
and produce quite large code changes.

s117
00:13:15.800 --> 00:13:20.000
Of course, the next step then becomes agents that are reviewing code.

s118
00:13:20.000 --> 00:13:31.800
Um and we we at Quarter believe that, you know, within the next 6 to 12 months, the majority of code that is being shipped will be reviewed uh not by human but by AI.

s119
00:13:31.800 --> 00:13:41.640
Um I think that's just a kind of natural consequence of the rate at which companies need to move, given that code review is not is now the bottleneck and I don't think we're going to accept that

s120
00:13:41.640 --> 00:13:44.080
for very long.

s121
00:13:44.080 --> 00:13:53.920
Um so really, right, our perspective at Corridor is around preventing vulnerabilities before the pull request, um as well as giving visibility into how AI coding tools are being used.

s122
00:13:53.920 --> 00:14:04.160
Um and I think that's really essential, right, is that security cannot be the blocker when it comes to companies accelerating their development, right?

s123
00:14:04.160 --> 00:14:07.360
Um acceleration is always going to win out.

s124
00:14:07.360 --> 00:14:16.240
Um so when we talk to security teams, the conversation is less around should you allow your, you know, development teams access to coding agents?

s125
00:14:16.240 --> 00:14:18.200
The answer is obviously yes, right?

s126
00:14:18.200 --> 00:14:21.840
It's more around how can you do that with guardrails in place, right?

s127
00:14:21.840 --> 00:14:33.000
Because what we're seeing is that without guardrails, yes, the coding agents can introduce vulnerabilities, um and in order to get to a point where, you know, development can be more autonomous,

s128
00:14:33.000 --> 00:14:47.839
that code can start to be reviewed um by AI and merged in without as much human oversight, we really need to have tooling in place that allows security teams to have that assurance,

s129
00:14:47.839 --> 00:14:52.320
um and to to give the blessing to their their engineering team to accelerate.

s130
00:14:52.320 --> 00:14:57.000
Um I I want to close with some of the policy perspective, right?

s131
00:14:57.000 --> 00:15:05.960
And um this is in part tied to the recent export controls on Mythos and Fable models, right?

s132
00:15:05.960 --> 00:15:16.920
As part of a letter uh led by Mikayla Gyalcsamis, um where we urged the White House to lift the export controls on these models.

s133
00:15:16.920 --> 00:15:25.240
And the perspective there is that the benefit to defenders far outweighs the risk, right?

s134
00:15:25.240 --> 00:15:35.120
These are very powerful and let's face it, right, dual-use models that can both be used to secure systems and also um to exploit them.

s135
00:15:35.120 --> 00:15:45.360
Uh to Anthropic's credit, they have done a lot of work with the Fable release, right, to have some safeguards in place such that um they're more skewed towards defenders

s136
00:15:45.360 --> 00:15:47.000
than than adversaries.

s137
00:15:47.000 --> 00:15:52.120
Um but this is also coming in the face of, right, increasingly powerful open-weight models.

s138
00:15:52.120 --> 00:16:08.280
Um you've probably seen, you know, documented um distillation attacks um where open-weight model providers can train on the output of um closed-weight models and that um as a result is quickly shrinking

s139
00:16:08.280 --> 00:16:15.600
the, you know, timeline between when a frontier closed-weight model comes out and when open-weight models catch up to that.

s140
00:16:15.600 --> 00:16:26.480
Um so whether we like it or not, right, adversaries already have access to incredibly powerful models um and they're already using them today to exploit systems, right?

s141
00:16:26.480 --> 00:16:40.400
So to me it becomes more question of how can we rapidly um get the kind of capabilities in the hands of defenders and I think that requires having these models be more widely available.

s142
00:16:40.400 --> 00:16:51.920
Um one cool thing I had the opportunity of doing a couple weeks ago was testifying to uh the US Congress um on the, you know, risks of both uh frontier models

s143
00:16:51.920 --> 00:16:53.880
as well as AI coding.

s144
00:16:53.880 --> 00:16:56.240
Um my recommendations had a couple elements, right?

s145
00:16:56.240 --> 00:16:59.960
One was to prevent vulnerabilities in new code going forward.

s146
00:16:59.960 --> 00:17:10.160
I think this is something that both every company as well as the US government should be focusing on and making sure that as development accelerates, right, security isn't being left behind there.

s147
00:17:10.160 --> 00:17:13.560
Um second is to harden the open-source foundation.

s148
00:17:13.560 --> 00:17:26.800
Um this is incredibly important, right, especially since open-source software is going to be the kind of proving ground for a lot of adversaries, right, who want to test out these models and exploit vulnerabilities they find

s149
00:17:26.800 --> 00:17:29.080
due to, right, the exact nature that's open-source.

s150
00:17:29.080 --> 00:17:36.480
So, so you can just go and run a very smart model on them and in all likelihood find many novel vulnerabilities.

s151
00:17:36.480 --> 00:17:43.320
Um so, so I think both the US government, but then private companies as well have a responsibility to help shore this up.

s152
00:17:43.320 --> 00:17:49.720
And like I mentioned, right, it's not just about one-off, um you know, vulnerability discoveries or patches.

s153
00:17:49.720 --> 00:18:02.800
It really has to be more systemic and start to get into um rewrites that can fundamentally reduce the risk of vulnerabilities that can be found whether by models today or in the future.

s154
00:18:02.800 --> 00:18:10.480
Um and then the last area of my recommendations were to foster an ecosystem of American weight made open-weight models, right?

s155
00:18:10.480 --> 00:18:18.040
I think in order for us to stay competitive here, it can't just be closed-weight models alone, right?

s156
00:18:18.040 --> 00:18:29.920
Um there's a number of reasons here, you know, one of those is that for many companies, while there's a place for for closed-weight models, you also might want to do things like fine-tuning models.

s157
00:18:29.920 --> 00:18:33.680
Um and that is only possible with an open-weight model.

s158
00:18:33.680 --> 00:18:41.040
Um so, so I think it is really essential, right, that um we have frontier open-weight models coming out of the United States.

s159
00:18:41.040 --> 00:18:50.520
We haven't seen as much of that to date, um but I think that's a critical element of American competitiveness when it comes to AI.

s160
00:18:50.520 --> 00:18:53.560
Um so, so that those were my overall recommendations, right?

s161
00:18:53.560 --> 00:18:58.320
And of course, all of this is in the context of these increasingly powerful models.

s162
00:18:58.320 --> 00:19:04.080
The hearing was, I believe, a few days before, you know, um Mythos and Fable came out.

s163
00:19:04.080 --> 00:19:09.880
Um and then, of course, all of the um export control um actions that were taken.

s164
00:19:09.880 --> 00:19:18.560
Um so, this is an incredibly rapidly evolving space, but I think that's why it's all the more important to go back to the basics, right?

s165
00:19:18.560 --> 00:19:26.560
What are the fundamental controls that can protect against um any vulnerabilities that can be discovered by models today or in the future?

s166
00:19:26.560 --> 00:19:35.160
And I think that's where we really ought to be spending our time using these models um to make our systems more resilient.

s167
00:19:35.160 --> 00:19:37.200
Um, so that's my talk.

s168
00:19:37.200 --> 00:19:38.320
Um, happy to be reached.

s169
00:19:38.320 --> 00:19:43.160
My email's here and thanks everyone for tuning in.