WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=imFedndyXYQ NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/imFedndyXYQ.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.400 --> 00:00:15.599 Hi, I'm Eugene U member of technical staff at Entropic. s3 00:00:15.599 --> 00:00:21.760 Um, I've spent the last several months with working with security teams and claw to find and fix vulnerabilities in codes and systems. s4 00:00:21.760 --> 00:00:24.320 So, I want to share with you what we've learned as much as we can. s5 00:00:24.320 --> 00:00:26.240 Before I get started, quick show of hands. s6 00:00:26.240 --> 00:00:31.039 How many of here here are security engineers in your day job? s7 00:00:31.039 --> 00:00:31.519 Okay. s8 00:00:31.519 --> 00:00:32.079 Okay. s9 00:00:32.079 --> 00:00:34.719 How many of you here are engineers in your day job? s10 00:00:34.719 --> 00:00:35.360 Oh, sweet. s11 00:00:35.360 --> 00:00:36.239 Okay, great. s12 00:00:36.239 --> 00:00:38.719 I have better understanding of how to give this talk now. s13 00:00:38.719 --> 00:00:40.320 Um, security engineers in the room. s14 00:00:40.320 --> 00:00:42.239 Please don't boo me if I oversimplify things. s15 00:00:42.239 --> 00:00:45.200 I want to get this message out to as many people as I can. s16 00:00:45.200 --> 00:00:50.079 So, first I want to share with you three highle trends that we are seeing in terms of model capabilities, right? s17 00:00:50.079 --> 00:00:54.000 Number of vulnerabilities found um and where the bottleneck is shifting to. s18 00:00:54.000 --> 00:01:03.920 Then I want to distill the lessons we've learned from working with dozens of uh organizations to improve their security posture as well as the bottleneck that you might encounter and how you can try to address it and prep for it. s19 00:01:03.920 --> 00:01:09.040 And finally, I want to share how you can get started this week with links to resources. s20 00:01:09.040 --> 00:01:12.080 So why do we care about so much about cyber security now? s21 00:01:12.080 --> 00:01:22.799 Um time horizon benchmarks track the length of time an AI model can complete a task measured against an actual human doing the task. s22 00:01:22.799 --> 00:01:26.159 Um so we all know about the meter eval task. s23 00:01:26.159 --> 00:01:28.080 You know how how much knowledge what a engine can do. s24 00:01:28.080 --> 00:01:32.159 The UK AI security institute has a cyber security version of this. s25 00:01:32.159 --> 00:01:40.320 You know these tasks require identifying exploiting security weaknesses in target systems and you know testing skills like reverse engineering and web exploitation. s26 00:01:40.320 --> 00:01:45.840 So in this chart we see that models are able to increasingly do longer cyber security task. s27 00:01:45.840 --> 00:01:51.119 Um but what's also interesting is that you kind of see a step jump against the previous regression line. s28 00:01:51.119 --> 00:01:55.439 So these models are a step improvement in capability. s29 00:01:55.439 --> 00:01:58.880 And what does this step improvement in capability buy us? s30 00:01:58.880 --> 00:02:04.079 Recently Mosilla Firefox published a number of security bug fixes they made each month. s31 00:02:04.079 --> 00:02:08.399 The average in 2025 was about 2020 20ish. s32 00:02:08.399 --> 00:02:13.440 And then you can see in February and March it kind of 3xed to about 60 and 70. s33 00:02:13.440 --> 00:02:18.560 And then in April it 7xed to 400. s34 00:02:18.560 --> 00:02:24.800 So what this means is that what's happened in April is 20x of last year's average. s35 00:02:24.800 --> 00:02:37.440 Um they attributed about twothirds of this to mess preview about 271 which shows that frontier models can help defenders like yourself find and fix vulnerabilities at scale. s36 00:02:37.519 --> 00:02:40.160 We also uh what are some notable vulnerabilities? s37 00:02:40.160 --> 00:02:40.800 Hands up. s38 00:02:40.800 --> 00:02:43.120 Who here remembers lock for shell? s39 00:02:43.120 --> 00:02:43.599 Anyone? s40 00:02:43.599 --> 00:02:47.680 Anyone was here over the December holidays like scrambling to patch this seu? s41 00:02:47.680 --> 00:02:51.920 So log for shell um is a bug in the Java logging library. s42 00:02:51.920 --> 00:02:57.040 If an attacker sends a string, you log it, the attacker can run code on your system. s43 00:02:57.040 --> 00:03:03.280 The Belgium defense ministry was breached within days and a fintech platform uh leaked 2 million users data. s44 00:03:03.280 --> 00:03:06.879 And then before that we had heartbleleed which is an a bug in open SSL. s45 00:03:06.879 --> 00:03:13.920 You know so much of the of the internet runs on OpenSSL and we we all know what impact it had. s46 00:03:13.920 --> 00:03:19.920 This is entropic sharing of our own work scanning more than a thousand open source repos. s47 00:03:19.920 --> 00:03:32.159 Um from 23,000 candidates uh 6,200 of them were rated as high or critical and at the time of the update 1,600 of them were reported to maintainers and about 100 patch upstream. s48 00:03:32.159 --> 00:03:39.599 We shared our observation that finding vulnerabilities now is quite straightforward. s49 00:03:39.599 --> 00:03:43.599 The bottleneck has now shifted to verification, triage, and patching. s50 00:03:43.599 --> 00:03:48.400 And I want to share with you everything we've learned about that so you can get ahead of it. s51 00:03:48.400 --> 00:03:52.000 So how do we do this in our own systems and code bases? s52 00:03:52.000 --> 00:03:54.159 Um, two words, agentic harnesses. s53 00:03:54.159 --> 00:03:58.720 I'm citing Mozilla again, and this was even before they worked with me preview. s54 00:03:58.720 --> 00:04:00.480 I'm going to read off the screen here. s55 00:04:00.480 --> 00:04:08.640 Early experiments showed that some promise, but the high rates of false positives made it impractical to scale. s56 00:04:08.640 --> 00:04:14.799 But the introduction of agentic harnesses that can reliably detect security issues has changed this. s57 00:04:14.799 --> 00:04:15.760 Right? s58 00:04:15.760 --> 00:04:23.680 These models and harnesses together can find real bugs and dismiss nonreproducible speculation. s59 00:04:23.680 --> 00:04:29.120 So the harness and the model working together can get a lot of work done. s60 00:04:29.120 --> 00:04:34.000 We found that most teams converge in approximately these six steps. s61 00:04:34.000 --> 00:04:37.199 And a big chunk of my talk is going to be about these six steps. s62 00:04:37.199 --> 00:04:37.360 Right? s63 00:04:37.360 --> 00:04:37.520 Right. s64 00:04:37.520 --> 00:04:38.800 The first two steps are set up, right? s65 00:04:38.800 --> 00:04:41.280 You do a bit of upfront investment per codebase. s66 00:04:41.280 --> 00:04:44.800 How many of you here know what a threat model is? s67 00:04:44.800 --> 00:04:45.280 Amazing. s68 00:04:45.280 --> 00:04:46.000 I love that. s69 00:04:46.000 --> 00:04:51.440 Uh a track model is some kind of like kind of a system design, but for your codebase or system, what are threat vectors? s70 00:04:51.440 --> 00:04:56.240 So, this helps you decide what are the vulnerabilities in your system that you care about. s71 00:04:56.240 --> 00:05:06.960 And then also, we also want an isolated sandbox where you let your model run possibly untrusted code or you let your model deliberately detonate proof of concepts of exploits. s72 00:05:06.960 --> 00:05:10.720 I'm going to call them pox to confirm that they are true positives. s73 00:05:10.720 --> 00:05:15.199 And then the next four are loop where you know we have discovery that identifies vulnerabilities. s74 00:05:15.199 --> 00:05:17.759 Verification confirms that they are real. s75 00:05:17.759 --> 00:05:18.800 We have triage. s76 00:05:18.800 --> 00:05:21.120 You know you can't give a developer 100 vulnerabilities. s77 00:05:21.120 --> 00:05:28.000 You got to identify the top 10 or 20 critical and high so they can focus their attention and then you can patch patch the bugs. s78 00:05:28.000 --> 00:05:30.400 So now I'm going to talk about each step. s79 00:05:30.400 --> 00:05:35.600 Now to make things concrete I want to share with you how they apply on a hypothetical system. s80 00:05:35.600 --> 00:05:37.600 So this is order service. s81 00:05:37.600 --> 00:05:41.360 Order service you type in a ID and then you look up an order. s82 00:05:41.360 --> 00:05:44.800 So what kind of v vulnerabilities might we find in this? s83 00:05:44.800 --> 00:05:47.440 Um the first step is the threat model. s84 00:05:47.440 --> 00:05:48.800 So why does this matter? s85 00:05:48.800 --> 00:05:58.000 So several teams if you look at a code right finding have found that having a well doumented thread model really increases your true positive rate to 90%. s86 00:05:58.000 --> 00:06:00.479 90% is outstanding if you ask me. s87 00:06:00.479 --> 00:06:02.639 I think anything above 75 is is great. s88 00:06:02.639 --> 00:06:04.160 That's where you want to aim for. s89 00:06:04.160 --> 00:06:11.520 Now to put it another way, one CISO said this, the model has great context of the code but poor context of the system. s90 00:06:11.520 --> 00:06:13.600 And why does this why does this happen? s91 00:06:13.600 --> 00:06:18.319 It's the whatever the model can read, the model will understand well. s92 00:06:18.319 --> 00:06:20.720 But there's so much information that the model cannot read. s93 00:06:20.720 --> 00:06:24.319 Why did the senior designer design the system this way? s94 00:06:24.319 --> 00:06:28.800 What are all the vulner vulnerabilities that on call has been fixing that has not been written down? s95 00:06:28.800 --> 00:06:37.440 All of this is now implicit context in your head, but you need to write it down somewhere, maybe in a threel to help the model be a better security engineer. s96 00:06:37.440 --> 00:06:39.840 So we can work with models to build these thread models. s97 00:06:39.840 --> 00:06:42.080 And the track model is really just a document, right? s98 00:06:42.080 --> 00:06:51.520 First, we can bootstrap a trap model, give the model access to all your documents and code, even past code commits, past patches that were made and what CVE they were patching s99 00:06:51.520 --> 00:06:56.960 and you can ask the model to infer, hey, what possible what other CVs have not been patched, right? s100 00:06:56.960 --> 00:06:58.240 So a model can do that. s101 00:06:58.240 --> 00:07:10.960 Then after the next step is that you should ask the model to interview the expert on the system right what are some things that could happen that we didn't plan for or what are some things that we actually don't s102 00:07:10.960 --> 00:07:24.720 need to worry about for vulnerability because maybe this is an internal VPN only app or there's a compensating control elsewhere a lot of this is not in the code right if you look at your codebase it's not there but it needs to be written down somewhere so the model can benefit for it s103 00:07:24.720 --> 00:07:34.160 so here's an example thread model of order service right you know the key asset is the data you know contains customer PII and the entry point is the orders API s104 00:07:34.160 --> 00:07:42.400 we can also see that the model has suggested some possible threat vectors like SQL injection and that the fact that you could call this API without any form of authentication s105 00:07:42.400 --> 00:07:54.720 then the second step is the sandbox so isolation and reproducibility in some cases we may be running untrusted code in some cases we may be asking the model to deliberately s106 00:07:54.720 --> 00:08:00.479 create exploits so we want some isolation ation um to prevent the model from doing bad things. s107 00:08:00.479 --> 00:08:13.280 So this isolation doing bad things could be data exfiltration or you know dropping things in production and of course uh you you want all of this to be running in a VM without egress and of of course without your cloud credentials. s108 00:08:13.280 --> 00:08:16.080 The second thing we want is reproducibility. s109 00:08:16.080 --> 00:08:27.199 So a lot of times uh you may find that the model may find a bug because you were using a old dependency or you want you want all of your agents to be starting from the same baseline container s110 00:08:27.199 --> 00:08:32.880 so that your agents can verif can can can run everything in a partners in the pipeline. s111 00:08:32.880 --> 00:08:38.479 So now building a repro uh representative sandbox of your system is actually quite difficult but the capability is quite useful. s112 00:08:38.479 --> 00:08:40.880 I think here's one team we worked with. s113 00:08:40.880 --> 00:08:52.880 They quoted the biggest ef the biggest lever we had is having the model test beds essentially sandboxes with live systems and where they can run and detonate the pox to confirm that they are true positives. s114 00:08:52.880 --> 00:08:56.800 So how would a sandbox for order service look like? s115 00:08:56.800 --> 00:09:00.959 Well this is a simple sandbox uh three docker images connected together. s116 00:09:00.959 --> 00:09:05.920 One for the app, one for the posgress database and one for the radius cache. s117 00:09:05.920 --> 00:09:13.600 And on the right we have the security agent uh which sits outside the target boundary trying to probe the app via HTTP. s118 00:09:13.760 --> 00:09:21.360 Now that we have the setup done we can now move on to doing the work of discovery finding the vulnerabilities. s119 00:09:21.360 --> 00:09:27.040 So the first is no so what are some there there are three things that matter a lot and there's no surprises here. s120 00:09:27.040 --> 00:09:29.440 The first thing is really context engineering. s121 00:09:29.440 --> 00:09:33.519 You're going to give the model all the context as much context as you can. s122 00:09:33.519 --> 00:09:36.959 If the if it's written down and if you share with the model, the model can find it. s123 00:09:36.959 --> 00:09:39.200 The model will be a better security engineer. s124 00:09:39.200 --> 00:09:41.440 You'll improve your vulnerability coverage. s125 00:09:41.440 --> 00:09:43.920 You'll reduce your false positive rate. s126 00:09:43.920 --> 00:09:47.200 The second is to have simple prop simpler props. s127 00:09:47.200 --> 00:09:54.399 Um, how many of here have found that as models get better, you actually need to remove from your prompts. s128 00:09:54.399 --> 00:10:00.399 That's what I found with every new model version of StepJum, I actually have to cut my prompt size by maybe about 50%. s129 00:10:00.399 --> 00:10:00.880 Right? s130 00:10:00.880 --> 00:10:10.880 So as models get better your prompts need to be less and less specific prescriptive even while you know previous models you might ask the model to look out for specific category classes s131 00:10:10.880 --> 00:10:20.000 but for newer models you can just probably say something like look for where untrusted data hits the trust boundary and the model is very good at inferring this. s132 00:10:20.000 --> 00:10:21.920 Finally give the model tools. s133 00:10:21.920 --> 00:10:24.560 So one example we had a pentesting team. s134 00:10:24.560 --> 00:10:32.880 They give the model tools to check the response to to query the API, check the response, read the logs, read the source code. s135 00:10:32.880 --> 00:10:36.959 Now this is just like the tools that you give a regular security engineer, right? s136 00:10:36.959 --> 00:10:39.600 A lot of times you expect the model to just read the code. s137 00:10:39.600 --> 00:10:40.560 That doesn't quite work. s138 00:10:40.560 --> 00:10:44.720 Allow the model to be dynamic to read the tools uh to to run the tools. s139 00:10:44.720 --> 00:10:48.800 And when they did this, their true positive rate was almost 100%. s140 00:10:48.800 --> 00:10:52.000 because the model could actually verify in the loop. s141 00:10:52.000 --> 00:10:58.079 So here is the get API for the order service and it's just five lines of code. s142 00:10:58.079 --> 00:11:01.279 Can anyone here spot the vulnerability? s143 00:11:01.279 --> 00:11:05.920 You can shout it out you guys. s144 00:11:05.920 --> 00:11:06.800 Exactly. s145 00:11:06.800 --> 00:11:13.120 So this is what the discovery agent flag right you know in line four we built a SQL query via Python string interpolation. s146 00:11:13.120 --> 00:11:20.560 So the user input input flows straight to the SQL query right and that means that someone can inject a SQL query that excfiltrates data from your database. s147 00:11:20.560 --> 00:11:34.079 This is code that I would have written right as me as a non security engineer it's just a very I just write the simplest code possible and this is how it would look like current models are very good at detect spotting simple stuff like this. s148 00:11:34.079 --> 00:11:42.240 So after discovery so we have maybe a thousand let's say 100 vulnerabilities we may have not been able to confirm them. s149 00:11:42.240 --> 00:11:44.000 The next step is verification. s150 00:11:44.000 --> 00:11:48.240 So if you think about machine learning, discovery optimizes for recall. s151 00:11:48.240 --> 00:11:51.440 Find as many vulnerabilities as possible. s152 00:11:51.440 --> 00:11:53.839 Verification optimizes for precision. s153 00:11:53.839 --> 00:12:00.000 Of the 100 vulnerabilities we found, how many of them are truly optim uh truly exploitable. s154 00:12:00.000 --> 00:12:02.560 So we have them both working separately. s155 00:12:02.560 --> 00:12:05.600 So we found that it's helpful to have these two steps separately, right? s156 00:12:05.600 --> 00:12:15.120 Because when the discovery agent is trying to verify its own work in the loop, trying to debate against itself in the loop, it may actually self censor and this may actually hurt recall. s157 00:12:15.120 --> 00:12:20.480 Um, and when building the verification agent, it's helpful for the verification agent to be independent and adversarial. s158 00:12:20.480 --> 00:12:28.320 Independent means that the verification agent doesn't see the reasoning traces, doesn't see all the work that the discovery agent has done. s159 00:12:28.320 --> 00:12:33.839 All the verification agent gets is the vulnerability and it approaches it from an adversarial point of view. s160 00:12:33.839 --> 00:12:38.560 assume that this vulnerability is false, try to confirm it's false or confirm it's true. s161 00:12:38.560 --> 00:12:44.480 So this sets the this sets a very high bar for the vulnerability which reduces the false positive rate. s162 00:12:44.480 --> 00:12:58.800 And also if you have invested the time in building building a representative sandbox um have the verification agent build the p detonate the pock to confirm if the ver the vulnerability is true. s163 00:12:58.800 --> 00:13:12.880 So in this case back to order service our verification agent is confirming the SQL injection right in a fresh container and this is why we have everything containers the verification agent is only given line number four here's where a SQL injection could happen s164 00:13:12.880 --> 00:13:21.760 and the verification agent actually tries just runs the corl command and the verification agent can see all the customer pi that has been exfiltrated. s165 00:13:21.760 --> 00:13:26.480 So now you have confirmation that a SQL injection can work. s166 00:13:26.800 --> 00:13:34.079 So after discovery and verification, we've maximized recall, we've optimized for precision, the next step is triage. s167 00:13:34.079 --> 00:13:38.639 So a lot of this may truly fire, but some of these are bug correctness issues. s168 00:13:38.639 --> 00:13:41.120 They have very low business impact. s169 00:13:41.120 --> 00:13:51.839 So across multiple teams, they shared that you can lose trust with product engineers by sending them all the vulnerabilities that are true, even those that are medium or low severity s170 00:13:51.839 --> 00:13:53.600 because those engineers can't cope. s171 00:13:53.600 --> 00:14:00.000 And as we've seen so many times, the scars resource now is engineer attention. s172 00:14:00.000 --> 00:14:01.600 So imagine you're a product engineer. s173 00:14:01.600 --> 00:14:05.120 What are the kinds of vulnerabilities you want to actually receive and fix? s174 00:14:05.120 --> 00:14:12.320 So as part of this step, we can do dduplication, you know, cheap heristic checks or we can get a model to help you with clustering and dduplication. s175 00:14:12.320 --> 00:14:15.519 We should also consider the severity of the exploit, right? s176 00:14:15.519 --> 00:14:20.160 Uh one thing to consider is the magnitude of the outcome of the impact. s177 00:14:20.160 --> 00:14:20.880 What is the magnitude? s178 00:14:20.880 --> 00:14:23.680 Is it a ring zero magnitude or ring one magnitude? s179 00:14:23.680 --> 00:14:25.839 Um, and also the likelihood of it occurring, right? s180 00:14:25.839 --> 00:14:31.440 How many hoops does an attacker have to jump through to actually make that exploit? s181 00:14:31.440 --> 00:14:33.760 And this is where the thread model is useful to all. s182 00:14:33.760 --> 00:14:36.480 You can think of the thread model as just synonymous with context. s183 00:14:36.480 --> 00:14:39.279 Just give the model context about your business, right? s184 00:14:39.279 --> 00:14:41.120 So for example, compensating controls. s185 00:14:41.120 --> 00:14:47.040 If you have a firewall, something that was initially high severity could actually be low. s186 00:14:47.040 --> 00:15:01.120 On the other hand, if you know that your database is mostly customer PI data or healthcare data, a model might have have set that as medium severity, but it actually could be high because the the contents of your data are so sensitive. s187 00:15:01.120 --> 00:15:06.800 So in this case, we have on the left triage that the agent has done. s188 00:15:06.800 --> 00:15:10.560 The agent has said that this is high severity for these reasons, right? s189 00:15:10.560 --> 00:15:12.560 High impact, high likelihood. s190 00:15:12.560 --> 00:15:26.160 Now on the right after human review the operator has confirmed that you know has downgraded this to low likelihood and the reason why it's low likelihood is because we have an application firewall that blocks SQL injections s191 00:15:26.160 --> 00:15:39.360 and also there's the fact that the service is internal only it's only for internal I warehousing or internal logistics it's not it's never internet facing therefore the likelihood is low now you can imagine like how would the model know this this is not written in a codebase s192 00:15:39.360 --> 00:15:52.000 it may be available in a system design it may be available in your documentation, it may be available in your confluence etc. So this is the kind of context you need to provide to the model so that the model can get better at this. s193 00:15:52.320 --> 00:16:00.720 Now after that we close the loop patching to validate the patch you know we can build a patch and then we want to validate there's a ladder of sorts s194 00:16:00.720 --> 00:16:15.040 first the original PC has to stop working that's basic second the existing test suite should stay green no regression and then we can go further right we can have a fresh discovery agent try to attack the patch code again is the patch comprehensive enough s195 00:16:15.040 --> 00:16:23.600 and teams have found that by giving the patching agent such feedback um you can actually the patch quality improves greatly right and you know This is the generative verifier loop. s196 00:16:23.600 --> 00:16:27.519 Give your agents context, give your agents feedback, and they they can do the do their work. s197 00:16:27.519 --> 00:16:31.199 And finally, a human confirms the patch before it's been merged. s198 00:16:31.199 --> 00:16:32.560 So here's how the patch will look like. s199 00:16:32.560 --> 00:16:41.759 You know, in the first div, we update the code, it's a oneline fix where we move the variable out of Python uh and therefore we reduce the code of SQL injection. s200 00:16:41.759 --> 00:16:46.720 And in the second div, not only do we update the code, not only do we fix the system, we close the loop. s201 00:16:46.720 --> 00:16:51.440 We make sure that the next time we run this loop, we run this scan, these issues don't happen again. s202 00:16:51.440 --> 00:16:54.560 And this is really important when you're building the building harnesses, right? s203 00:16:54.560 --> 00:16:55.680 You're building loops. s204 00:16:55.680 --> 00:17:00.240 They're operational expense, but when you close the loop, they now become capital expense. s205 00:17:00.240 --> 00:17:03.199 You get better with each iteration you run. s206 00:17:03.199 --> 00:17:04.720 So you make clear the compensity controls. s207 00:17:04.720 --> 00:17:08.959 We have the application firewall and the fact that it's internal only. s208 00:17:08.959 --> 00:17:09.199 Okay. s209 00:17:09.199 --> 00:17:14.000 So the quick recap, these are the six steps and each step focuses on one thing. s210 00:17:14.000 --> 00:17:16.480 Track model provides context. s211 00:17:16.480 --> 00:17:19.360 Sandbox provides isolation and reproducibility. s212 00:17:19.360 --> 00:17:21.839 This is the baseline fundamentals that you need to have. s213 00:17:21.839 --> 00:17:24.240 This will carry throughout every scan. s214 00:17:24.240 --> 00:17:27.120 You have context and reproducibility. s215 00:17:27.120 --> 00:17:33.039 Then after that, the rest is really if you look at it, it's if you if you squint, it actually looks like an ML pipeline. s216 00:17:33.039 --> 00:17:38.160 Recall, precision, ranking, and closing the loop. s217 00:17:38.160 --> 00:17:41.520 It's a regular machine learning pipeline. s218 00:17:43.200 --> 00:17:45.840 Now, that's the technical challenge we have. s219 00:17:45.840 --> 00:17:47.360 There's another technical challenge we have. s220 00:17:47.360 --> 00:17:54.080 And as my previous director would say, non-technical problems are an order of magnitude harder than technical problems. s221 00:17:54.080 --> 00:17:57.360 So what are some other bottlenecks you might have? s222 00:17:57.360 --> 00:17:59.919 So imagine you have a process. s223 00:17:59.919 --> 00:18:04.160 What happens when you have one or two orders of magnitude more input? s224 00:18:04.160 --> 00:18:05.919 What is going to start to break? s225 00:18:05.919 --> 00:18:09.679 In the case of the harness, the scanning harness, you just have more engineering. s226 00:18:09.679 --> 00:18:11.039 You spend more compute. s227 00:18:11.039 --> 00:18:12.400 You pay more money. s228 00:18:12.400 --> 00:18:15.440 Things that can be solved with money are not really problems. s229 00:18:15.440 --> 00:18:18.480 But human attention doesn't scale. s230 00:18:18.480 --> 00:18:26.960 Your deaf, your product engineers and your security engineers, what if they don't agree on what high severity or uh critical severity is? s231 00:18:26.960 --> 00:18:28.160 You need to put them in a room. s232 00:18:28.160 --> 00:18:31.360 You need to write down all these rules so that everyone agrees. s233 00:18:31.360 --> 00:18:35.280 And you know your threat models, they're all stuck in people's heads right now. s234 00:18:35.280 --> 00:18:43.520 Someone needs they need to be interviewed maybe by a model or by someone else and write all that trap model down so that now you can have agents help you. s235 00:18:43.520 --> 00:18:46.720 And finally, you know, patching patch patch review. s236 00:18:46.720 --> 00:18:54.960 Um, I still don't know of many companies that gone through fully automated patch review, especially when p uh patching security issues. s237 00:18:54.960 --> 00:18:59.360 So, here are some organizational bottlenecks that you might hit and what you can do. s238 00:18:59.360 --> 00:19:01.440 The first thing is vulnerability routing. s239 00:19:01.440 --> 00:19:05.840 So, if you maybe get a dozen vulnerabilities a month, you can probably curate them. s240 00:19:05.840 --> 00:19:08.720 You know, email them to specific teams, assign Jira tickets. s241 00:19:08.720 --> 00:19:11.360 When you have hundreds, you can't do this. s242 00:19:11.360 --> 00:19:20.480 And this can really be as simple as you know sending it to the code owner etc. um or or the service owner and it doesn't really need an LM in the loop. s243 00:19:20.480 --> 00:19:22.799 Then second thing is severity calibration. s244 00:19:22.799 --> 00:19:32.720 The the one hard thing we found is that product engineers and security engineers red team blue team they may not agree on what high severity is and all this requires a lot of your business context. s245 00:19:32.720 --> 00:19:42.480 I really think it helps to have everyone discuss and write it all down once and for all agree upon that that you can send that to agent and finally the patching bandwidth. s246 00:19:42.480 --> 00:19:54.160 Um I'm sure no one writes very few of you here write code by hand now but it's also very challenging for you to just be given the vulnerability and try to implement a patch even with LM assistance. s247 00:19:54.160 --> 00:20:02.799 you definitely want to be moving towards AI generated patches and then have the human loop verify and you know generate the patch reattack the patch. s248 00:20:02.799 --> 00:20:09.840 Okay, if you only remember one thing well here are the three things you want I want you to remember. s249 00:20:09.840 --> 00:20:17.440 I would like to you start now please start with open source dependencies start with whatever you're comfortable with. s250 00:20:17.440 --> 00:20:20.400 Two I want to teach you how to climb the learning curve. s251 00:20:20.400 --> 00:20:23.200 Don't try to aim for automation immediately. s252 00:20:23.200 --> 00:20:23.679 Right? s253 00:20:23.679 --> 00:20:25.520 Start interactively. s254 00:20:25.520 --> 00:20:28.960 Do it hands on the wheel with uh claw code or your favorite ID. s255 00:20:28.960 --> 00:20:29.840 Doesn't matter. s256 00:20:29.840 --> 00:20:31.039 Learn where you get cut. s257 00:20:31.039 --> 00:20:32.480 Learn what kind of context you're missing. s258 00:20:32.480 --> 00:20:34.400 Learn where precision is low. s259 00:20:34.400 --> 00:20:39.120 Then third is don't just aim for scanning. s260 00:20:39.120 --> 00:20:41.520 Uh scanning is not the bottleneck. s261 00:20:41.520 --> 00:20:42.960 Verification, triage, and patching. s262 00:20:42.960 --> 00:20:44.480 And then all your organizational processes. s263 00:20:44.480 --> 00:20:45.919 Those are the bottleneck. s264 00:20:45.919 --> 00:20:46.320 Over there. s265 00:20:46.320 --> 00:20:47.440 I have a couple of resources. s266 00:20:47.440 --> 00:20:51.360 We have cloud security to scan your codebase, validate findings such as patches. s267 00:20:51.360 --> 00:20:52.320 We have a blog. s268 00:20:52.320 --> 00:21:08.880 We have open source repos for with that includes the interactive skills and autonomous harnesses and all you have to do is to run this few steps to very quickly get hands on the wheel and in it if you look at step five you have the harness and you can just customize it. s269 00:21:08.880 --> 00:21:09.840 That's all I had. s270 00:21:09.840 --> 00:21:12.000 Thank you.