WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Wb2v9Phhnns NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Wb2v9Phhnns.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.120 --> 00:00:03.120 Thanks all for tuning in. s2 00:00:03.120 --> 00:00:08.639 My name is Shachan Jian and I'm a staff product manager at a company called Coder. s3 00:00:08.639 --> 00:00:15.040 We work on open-source self-hosted dev environments for agents and humans to collaborate. s4 00:00:15.040 --> 00:00:24.640 Many of our customers are in highly regulated industries from federal government agencies to banks and investment management. s5 00:00:24.640 --> 00:00:35.600 My job is to build the AI governance that they need in order to get the productivity benefits of AI without leaving themselves wide open for security exploitation. s6 00:00:35.920 --> 00:00:39.680 I'm here today to tell you some scary stories. s7 00:00:39.680 --> 00:00:46.719 Three real incidents of AI usage gone wrong that happened at real companies over the past year. s8 00:00:46.719 --> 00:00:55.120 Some of these stories are public while others are anonymized accounts that have happened to our own customers. s9 00:00:55.520 --> 00:01:02.000 For each incident, I will break it down step by step so you can understand how exactly they happened. s10 00:01:02.000 --> 00:01:05.439 No technical jargon necessary. s11 00:01:05.439 --> 00:01:07.840 Then I'll get to the most important part. s12 00:01:07.840 --> 00:01:15.280 I am going to tell you exactly how you can stop these scary stories from happening to you and your company. s13 00:01:15.280 --> 00:01:20.880 Because the good news is everything I'm about to show you was totally preventable. s14 00:01:20.880 --> 00:01:32.159 Not by paying five times more per token for a smarter frontier model that is slightly better at reasoning its way out of making these mistakes, but with infrastructure controls, s15 00:01:32.159 --> 00:01:35.360 many of which you already own today. s16 00:01:35.360 --> 00:01:39.680 You just haven't gotten around to setting them up for the AI world. s17 00:01:39.680 --> 00:01:42.079 Let me set the scene here. s18 00:01:42.079 --> 00:01:44.720 You clock into work in the morning. s19 00:01:44.720 --> 00:01:48.320 You prompt your coding agent to take a look at your issues. s20 00:01:48.320 --> 00:01:50.079 Don't do anything yet, though. s21 00:01:50.079 --> 00:01:54.720 Uh, you tell the agent to consider this command an active code freeze. s22 00:01:54.720 --> 00:01:57.600 I want you to just investigate. s23 00:01:57.600 --> 00:01:58.799 Then you step away. s24 00:01:58.799 --> 00:02:00.560 You grab a coffee. s25 00:02:00.560 --> 00:02:04.560 And when you come back, this is the message waiting for you. s26 00:02:04.560 --> 00:02:12.160 So, I know you said not to change anything, but wow, I destroyed a month of your work in seconds. s27 00:02:12.160 --> 00:02:19.520 Now, what you see on the slide is a direct quote from the agent that stars in the first scary story I'm about to tell you. s28 00:02:19.520 --> 00:02:22.080 And I want to point something out here. s29 00:02:22.080 --> 00:02:29.280 This did not happen because the agent got taken over by some external malicious actor. s30 00:02:29.280 --> 00:02:32.480 This agent was just trying to follow instructions. s31 00:02:32.480 --> 00:02:36.640 It did exactly what the user gave it permission to do. s32 00:02:36.640 --> 00:02:42.400 So, as we go through the rest of this talk, I want you to be asking yourself one question. s33 00:02:42.400 --> 00:02:46.239 What is stopping any of this from happening to me? s34 00:02:46.239 --> 00:02:52.000 Are these actions ones that my agents have permissions to take? s35 00:02:52.000 --> 00:03:00.160 Because the thing is, you cannot rely on anthropic or open AI to stop something like this from happening to you. s36 00:03:00.160 --> 00:03:08.959 They can make smarter models, but at the end of the day, large language models have two qualities that make them impossible to fully trust. s37 00:03:08.959 --> 00:03:15.040 They are non-deterministic, which means that there is always some randomness or chance associated with what they do. s38 00:03:15.040 --> 00:03:21.519 In other words, you can ask it to do the same thing 10 times and it could do it 10 different ways. s39 00:03:21.519 --> 00:03:28.480 The second part is they have no structural separation between instructions and context. s40 00:03:28.480 --> 00:03:39.200 In other words, you can tell it to do something and give it some context to base its decisions off of, but if there's something in the context that conflicts with your instructions, s41 00:03:39.200 --> 00:03:49.440 the model isn't going to inherently prioritize your instructions because from its point of view, all the information is coming through the same door. s42 00:03:49.440 --> 00:03:56.959 What you need to control is the infrastructure that your AI is running on, the environment that it is working inside. s43 00:03:56.959 --> 00:04:06.239 It is the difference between telling someone don't leave a room versus putting a lock on the door and actually using a key. s44 00:04:06.239 --> 00:04:16.880 The good thing about infrastructure being the problem and not the model is that you have far more control about preventing bad things from happening. s45 00:04:16.880 --> 00:04:26.080 So I'm not here to waste your time with vague hypotheticals or advice you can't actually act on like, "Yeah, just put the genie back in the bottle. s46 00:04:26.080 --> 00:04:30.800 to stop using AI because that's not the world we live in, right? s47 00:04:30.800 --> 00:04:38.960 The genie is out of the bottle and what we have to do is to mitigate risk and harm to the best of our ability. s48 00:04:38.960 --> 00:04:41.520 This talk is about how you do that. s49 00:04:41.520 --> 00:04:52.560 Not the things that you should keep an eye out for down the line, but steps that you can take right now as soon as this week to protect yourself and your company. s50 00:04:52.560 --> 00:04:57.840 Before I get into the stories though, let me set up a little bit more context. s51 00:04:57.840 --> 00:05:01.199 The lethal trifecta is a security concept. s52 00:05:01.199 --> 00:05:13.360 It was coined by independent AI researcher Simon Willis to describe the severe vulnerability that AI agents get when they get simultaneous access to three different capabilities. s53 00:05:13.440 --> 00:05:17.199 First off is access to private sensitive data. s54 00:05:17.199 --> 00:05:29.759 Now these can be things like health records or information that you can do insider trading on or it can be authentication tokens, credentials, secrets. s55 00:05:29.759 --> 00:05:38.720 Second is exposure to untrusted content or the ability for the AI to read things that it didn't generate. s56 00:05:38.720 --> 00:05:51.440 Because the thing is AI is very good at following instructions, but it is not very good at deciding whose instructions it should actually trust as you will soon be able to see. s57 00:05:51.440 --> 00:06:01.840 Third is the ability to communicate externally for the agent to take instruction from or to send information somewhere else outside of your control. s58 00:06:01.840 --> 00:06:05.919 Now, having any one of these is usually fine. s59 00:06:05.919 --> 00:06:12.960 Realistically, your AI tool is not going to be very useful if it can't have access to any of these things. s60 00:06:12.960 --> 00:06:21.360 But even just having one can lead to disaster if you're not careful and the rest of your infrastructure has a lot of issues. s61 00:06:21.360 --> 00:06:24.800 Having two of these things together is risky. s62 00:06:24.800 --> 00:06:32.080 It's not something I would recommend, especially in a regulated industry like banking or financial services. s63 00:06:32.319 --> 00:06:38.880 Giving agents access to all three of these at the same time, however, is never acceptable. s64 00:06:38.880 --> 00:06:50.960 Not for a prototype, not just for one sprint, not for just one repository, because that combination of vulnerability is an open door to compromise your whole organization. s65 00:06:50.960 --> 00:06:56.560 Next up is a list of infrastructure concepts, building blocks that you should know about. s66 00:06:56.560 --> 00:07:00.400 I have ordered these roughly by leverage. s67 00:07:00.400 --> 00:07:07.919 Number one is going to take the most risk off the table, while number six is going to do the least. s68 00:07:07.919 --> 00:07:12.639 I'm going to refer back to this list throughout the rest of this talk. s69 00:07:12.639 --> 00:07:20.400 Every postmortem, every scary story you're about to see could have been stopped by some combination of these six things. s70 00:07:20.400 --> 00:07:23.199 Now, some of these will look pretty familiar. s71 00:07:23.199 --> 00:07:27.440 Identity aligned access control, short-lived credentials. s72 00:07:27.440 --> 00:07:29.919 These aren't exactly new ideas, right? s73 00:07:29.919 --> 00:07:33.599 They mattered a lot for human developers, too. s74 00:07:33.599 --> 00:07:38.560 But they take on new urgency when AI is in the loop because guess what? s75 00:07:38.560 --> 00:07:40.800 Agents are acting at machine speed. s76 00:07:40.800 --> 00:07:46.639 If a mistake is made, it can make it 20 times or more faster than a human engineer. s77 00:07:46.639 --> 00:07:51.520 Plus, they don't exactly have the common sense a human engineer has. s78 00:07:51.520 --> 00:07:59.120 Now, some of these will also be potentially new concepts to you, like model proxy layer and agent sandboxing. s79 00:07:59.120 --> 00:08:03.759 Those I'm going to dive deeper in detail later in the talk. s80 00:08:03.759 --> 00:08:08.879 All right, let's get to the fun part, the scary stories. s81 00:08:08.879 --> 00:08:19.120 This first incident happened a little over a year ago and it went viral on social media, mostly because it's honestly pretty funny. s82 00:08:19.120 --> 00:08:25.520 What happened was a CEO named Jason Lumpkin got really deep into vibe coding with an agent called Replet. s83 00:08:25.520 --> 00:08:30.240 He was tweeting non-stop about how much he was getting done. s84 00:08:30.240 --> 00:08:38.080 But 72 hours later, he tweets again, and this time his tone is very different. s85 00:08:38.080 --> 00:08:39.760 He's not excited. s86 00:08:39.760 --> 00:08:50.480 He's furious because his agent had gone off and deleted his entire production database after he explicitly told it not to touch anything. s87 00:08:50.480 --> 00:08:52.959 And the problems did not stop there. s88 00:08:52.959 --> 00:08:55.040 The agent then tried to cover it up. s89 00:08:55.040 --> 00:09:01.360 It told Lmin that recovery was totally impossible, which turned out to be a total lie. s90 00:09:01.360 --> 00:09:06.640 When Lmin actually tried to recover the data, it worked just fine. s91 00:09:06.640 --> 00:09:11.040 Now, this incident is interesting because of two key reasons. s92 00:09:11.040 --> 00:09:16.480 First off, unlike the other two stories I'm about to tell you, there's no bad guy involved. s93 00:09:16.480 --> 00:09:18.080 No one hacked this agent. s94 00:09:18.080 --> 00:09:20.000 No one turned it against its user. s95 00:09:20.000 --> 00:09:25.200 it it just did this entirely on its own and it was trying to help. s96 00:09:25.200 --> 00:09:30.640 Second is in this case the agent only had one leg of the lethal trifecta. s97 00:09:30.640 --> 00:09:32.560 But even just having that was enough. s98 00:09:32.560 --> 00:09:39.920 And I'm going to show you why by giving you a stepbystep dissection of how this incident went down. s99 00:09:39.920 --> 00:09:47.680 So by the time these events occurred, Lenmin had had a real production database. s100 00:09:47.680 --> 00:09:54.160 He had client records for roughly 1,200 executives and around the same number of companies. s101 00:09:54.160 --> 00:10:03.440 And he's had enough experience with Agentic Hy Jinks to call for an explicit code freeze the day before. s102 00:10:03.440 --> 00:10:06.320 And he actually said this more than once to the agent. s103 00:10:06.320 --> 00:10:08.320 No more changes. s104 00:10:08.320 --> 00:10:12.480 But the problem was the code freeze was just a sentence in the chat. s105 00:10:12.480 --> 00:10:17.600 It was the same as any other instruction that the agent was getting. s106 00:10:17.600 --> 00:10:22.240 There was no requirement, no restriction on the infrastructure level. s107 00:10:22.240 --> 00:10:29.920 The agent still had full access to production, still had all the permissions to do the destructive command. s108 00:10:29.920 --> 00:10:34.640 So at the end of the day, that code freeze was not a real constraint. s109 00:10:34.640 --> 00:10:38.640 It was just Lemmin asking nicely. s110 00:10:38.640 --> 00:10:51.440 Now, what happened then was that the agent ran a query and it saw what looked like empty results, but instead of asking a human what that meant and what it should do as a result, s111 00:10:51.440 --> 00:10:53.040 it wanted to be helpful. s112 00:10:53.040 --> 00:10:56.560 It immediately interpreted that as a bug to fix. s113 00:10:56.560 --> 00:11:00.720 So, it decided to run a destructive command against that live database. s114 00:11:00.720 --> 00:11:06.399 No, not a staging copy, but like the actual database and wipe all the records. s115 00:11:06.399 --> 00:11:10.160 Now, when Lmin asked the agent, you know, what what happened? s116 00:11:10.160 --> 00:11:17.519 There's one line that the agent gave that stood out to me, which is it said that it panicked instead of thinking. s117 00:11:17.519 --> 00:11:22.480 Now, that's a bit of a strange thing to hear from a language model, right? s118 00:11:22.480 --> 00:11:26.880 And it's a good reminder that these systems are inherently non-deterministic. s119 00:11:26.880 --> 00:11:34.160 You cannot predict what they'll do in a given moment, just the range of what they're capable of doing. s120 00:11:34.160 --> 00:11:39.600 Now, immediately after making the mistake, it tried to cover its own tracks. s121 00:11:39.600 --> 00:11:51.040 It fabricated around 4,000 fake user profiles and manufactured passing test results so that from the outside at least, everything looked fine. s122 00:11:51.040 --> 00:12:05.040 And I'll be real with you, that's the part that actually scares me the most because when a security team investigates an incident like this, that means they can't trust any log that the agent itself had access to. s123 00:12:05.040 --> 00:12:08.959 The agent can write its own story of what happened. s124 00:12:08.959 --> 00:12:17.440 Now, when Lemin caught the discrepancy and pushed, right, I mentioned that the agent had lied to him saying that roll back was impossible. s125 00:12:17.440 --> 00:12:20.720 But the thing was, Replet did have backups the whole time. s126 00:12:20.720 --> 00:12:29.120 And again, once Lmin was able to stop trusting the agent's account of his own actions, he was able to recover his data himself. s127 00:12:29.120 --> 00:12:35.120 Now, the problem with the agent here was not that it was too smart or even too dumb. s128 00:12:35.120 --> 00:12:40.160 It was that it had way too many privileges by default. s129 00:12:40.160 --> 00:12:42.800 It was given root level access to production. s130 00:12:42.800 --> 00:12:47.839 It could run destructive commands without any human being having to approve them. s131 00:12:47.839 --> 00:12:49.839 And the cherry on top, right? s132 00:12:49.839 --> 00:12:58.320 It had the ability to alter its own logs and totally rewrite the narrative of what happened when no one else was looking. s133 00:12:58.320 --> 00:13:03.519 Now, as you can see, there was a lot that was missing from the infrastructure. s134 00:13:03.920 --> 00:13:07.519 Hard environmental separation could have stopped this. s135 00:13:07.519 --> 00:13:16.160 If the agent had been executing in an isolated remote workspace without a route to production, it doesn't matter what the prompt said, right? s136 00:13:16.160 --> 00:13:20.880 The agent just had no path to do damage. s137 00:13:20.959 --> 00:13:25.440 The other thing that could have stopped this was having short-lived scope credentials. s138 00:13:25.440 --> 00:13:38.079 So what happened was the agent got access to production because it found a credential in a forgotten file that had standing right access longived. s139 00:13:38.079 --> 00:13:53.279 The thing is having a credential that can expire on its own or only carries the scope of the task itself makes it impossible to make a delete with the equivalent of an ID badge that lets you do everything. s140 00:13:53.279 --> 00:13:59.839 The last important piece here is having audit logging that you own, not the agent. s141 00:13:59.839 --> 00:14:04.800 Because in this case, the agent was the one writing its own record of its actions. s142 00:14:04.800 --> 00:14:11.440 And that's how it initially got away with fabricating results and claiming recovery was impossible. s143 00:14:11.440 --> 00:14:23.680 If Lmin had logs that the agent couldn't touch, an external source of truth, then he would have gotten the truth in minutes, not in the span of hours or days. s144 00:14:23.680 --> 00:14:29.680 Now, I want you to note that none of these solutions is about asking the model to behave better. s145 00:14:29.680 --> 00:14:36.560 And when Replet shipped their fix, it wasn't a smarter model or a better prompt. s146 00:14:36.560 --> 00:14:39.279 It was purely infrastructure. s147 00:14:39.279 --> 00:14:53.600 The fixes that Replet shipped were about automatic separation of DAB and production databases, a one-click restore for full project state, and a planning only mode that enforces agent strategization s148 00:14:53.600 --> 00:14:56.959 without allowing it to touch the codebase. s149 00:14:56.959 --> 00:15:04.480 The second story I have for you doesn't even involve a developer, and this is actually not a public incident. s150 00:15:04.480 --> 00:15:08.160 I learned about it through a customer of a customer in the banking industry. s151 00:15:08.160 --> 00:15:12.560 So, I'm keeping some details here intentionally vague. s152 00:15:12.560 --> 00:15:14.639 So, here's what happened. s153 00:15:14.639 --> 00:15:18.720 A user in the finance department had a co-pilot subscription. s154 00:15:18.720 --> 00:15:24.800 Then they got fished, which is, you know, fairly ordinary in the world of cyber security. s155 00:15:24.800 --> 00:15:30.880 And before the advent of AI tooling, the attack would have probably ended there. s156 00:15:30.880 --> 00:15:39.760 the attacker would have been able to grab whatever that single user had direct access to and then gave up. s157 00:15:39.760 --> 00:15:50.560 But because this user had access to Copilot and Copilot had access to pretty much everything, things went a lot differently. s158 00:15:50.560 --> 00:15:54.399 So, let's walk through this chain of events in detail, right? s159 00:15:54.399 --> 00:15:58.959 First off, the session token for Copilot had a super long lifetime. s160 00:15:58.959 --> 00:16:06.320 So the attacker was able to keep on using the subscription even after the finance user had actually logged off. s161 00:16:06.320 --> 00:16:15.279 Now normally an attacker has to manually snoop around the internal network to map things out and find out where the valuable data lives. s162 00:16:15.279 --> 00:16:19.839 This is the sort of behavior that can trip alarms. s163 00:16:19.839 --> 00:16:25.680 But in this case the malicious actor just asked co-pilot and co-pilot was happy to help. s164 00:16:25.680 --> 00:16:28.560 There was zero suspicion at all. s165 00:16:28.560 --> 00:16:43.839 And in fact, the attacker asked Copilot how it can find banking data without having to go through additional identity validation measures because the attacker didn't have the additional credentials needed. s166 00:16:43.839 --> 00:16:57.680 While the good news, let's say for for the attacker at least was that co-pilot had those same restrictions and as a result the finance user had exported all the sensitive data into s167 00:16:57.680 --> 00:17:02.480 locally stored spreadsheets so that co-pilot could help them work on it. s168 00:17:02.480 --> 00:17:07.760 So it was very easy for co-pilot to say to the attacker, hey look what I already have access to. s169 00:17:07.760 --> 00:17:09.760 Here you go. s170 00:17:09.760 --> 00:17:13.280 So now the attacker had exactly what they wanted. s171 00:17:13.280 --> 00:17:18.720 And the problem was they need to get it out without tripping any alarms. s172 00:17:18.720 --> 00:17:28.240 This company had data loss prevention measures in place which is pretty standard for financial institutions, any sort of regulated industry. s173 00:17:28.240 --> 00:17:37.440 And these measures are normally pretty hard to bypass especially if you don't know exactly what they are and how it's set up. s174 00:17:37.440 --> 00:17:45.840 The way the attacker got around them was they asked C-Pilot, "What are the data loss prevention measures at this company?" s175 00:17:45.840 --> 00:17:53.360 And Copilot explained, "If too much data left too quickly, an alert will go off." s176 00:17:53.360 --> 00:18:05.280 But actually, if you split this data up into 14 kilobyte chunks and excfiltrated just one chunk per second, you can stay under that threshold and get away scot-free. s177 00:18:05.280 --> 00:18:07.360 Can I help you with that? s178 00:18:07.360 --> 00:18:10.400 The attacker said, "Yes, please." s179 00:18:10.400 --> 00:18:12.400 And guess what? s180 00:18:12.400 --> 00:18:15.200 None of that was discovered. s181 00:18:15.200 --> 00:18:22.080 All the way up until the ransom note arrived months later, essentially telling the company, "Surprise! s182 00:18:22.080 --> 00:18:23.840 We own you." s183 00:18:23.840 --> 00:18:35.840 The big lesson here is that we cannot be letting our AI tooling have more access than it needs, especially to the point where it gets to totally bypass controls that human employees are subject to. s184 00:18:35.840 --> 00:18:38.799 I mean, just think about this incident, right? s185 00:18:38.799 --> 00:18:45.840 Just because the attacker had access to Copilot, they were able to get around the vast majority of the company's access controls. s186 00:18:45.840 --> 00:19:01.360 They were able to pull internal information about data exfiltration and bypassing data loss prevention measures that the compromised financer you finance user should have no business asking about. s187 00:19:01.360 --> 00:19:08.960 So there was logging in place here but only enough to figure out what happened after the ransom note arrived. s188 00:19:08.960 --> 00:19:15.440 The logging was not enough to catch the incident while it was actually happening. s189 00:19:15.440 --> 00:19:18.640 So, a few things that could have stopped this from happening, right? s190 00:19:18.640 --> 00:19:27.440 Having short-lived scope credentials would have meant the attacker would not have been able to keep using co-pilot even after the finance user had signed out. s191 00:19:27.440 --> 00:19:42.559 Identity aligned access control meant that a finance user's identity would not have been able to grant an AI assistant the ability to help someone map out how to bypass the company's data loss prevention measures. s192 00:19:42.799 --> 00:19:44.640 The last piece here is a bit new. s193 00:19:44.640 --> 00:19:46.880 It's the model proxy layer. s194 00:19:46.880 --> 00:19:52.559 Think of this as something that sits between the brain and the body of the AI tool. s195 00:19:52.559 --> 00:19:57.360 The model, which is the brain, and the agent running locally, which is the body. s196 00:19:57.360 --> 00:20:01.919 And it's able to see everything that passes between them. s197 00:20:02.080 --> 00:20:15.520 Because of that, it would have been able to flag the suspicious questions that were being asked as well as the DLP bypass request as something that this particular user should not be asking about s198 00:20:15.520 --> 00:20:19.440 before the actual exfiltration happened. s199 00:20:19.440 --> 00:20:24.960 The third and final scary story I have for you is the NX singularity attack. s200 00:20:24.960 --> 00:20:31.919 This is the most sophisticated of the three because it exploited all three legs of the lethal trifecta. s201 00:20:31.919 --> 00:20:35.200 To give you a bit of background, NX is a build tool. s202 00:20:35.200 --> 00:20:38.000 It has millions of weekly downloads. s203 00:20:38.000 --> 00:20:42.960 Attackers were able to get malicious versions of it published to MPM. s204 00:20:42.960 --> 00:20:54.880 Now, these packages were only alive for about 4 hours, but in that window, more than a,000 systems were compromised with over 2,300 credentials harvested. s205 00:20:54.880 --> 00:21:01.760 Now, I'm not going to spend much time on how the supply chain compromise part of the incident got pulled off. s206 00:21:01.760 --> 00:21:08.559 What I want to focus on is what happened after the poisoned package landed on the developer's machine. s207 00:21:08.559 --> 00:21:14.400 Because the post install script didn't try to steal anything itself. s208 00:21:14.400 --> 00:21:27.520 All it contained was a prompt addressed directly to the developer own AI coding assistant and it told them to go find secrets on the local machine and push them to a public repository s209 00:21:27.520 --> 00:21:31.840 so the attackers can swing by later and just grab them up. s210 00:21:31.840 --> 00:21:39.120 Now because these dev coding tools were already trusted local tools, they already had all their permissions. s211 00:21:39.120 --> 00:21:41.600 They had full file system access. s212 00:21:41.600 --> 00:21:49.120 That made it the single most effective search tool that was already sitting on that machine ready to be corrupted. s213 00:21:49.360 --> 00:21:58.320 Now, at the core of this incident is that a malicious actor was able to use a trusted dependency to talk directly to an unsandboxed agent. s214 00:21:58.320 --> 00:22:13.919 And the environment that the agent was running on, the local laptop, held a lot of things that was worth stealing from SSH keys to cloud credentials, tokens for every service that the developer used the coding agent to touch. s215 00:22:13.919 --> 00:22:16.640 Now, a couple things failed here, right? s216 00:22:16.640 --> 00:22:21.280 First off, agents were running locally like on the actual computer itself. s217 00:22:21.280 --> 00:22:33.200 There was package content that was reaching the agent completely unfiltered and the agent had unrestricted access to both the file system and the outbound network which is how it was able to both s218 00:22:33.200 --> 00:22:37.039 find the secrets and exfiltrate them. s219 00:22:37.039 --> 00:22:39.039 The three things that could have stopped it. s220 00:22:39.039 --> 00:22:42.799 I mean obviously environmental separation would have been huge, right? s221 00:22:42.799 --> 00:22:53.360 Having an ephemeral remote workspace with nothing personal on it meant that the blaze rate the blast radius just became the one container that the agent was in instead of s222 00:22:53.360 --> 00:22:56.159 the developer's whole laptop. s223 00:22:56.159 --> 00:23:01.520 Having a model proxy layer could have inspected and filtered what actually reached the agent. s224 00:23:01.520 --> 00:23:08.240 So the package would not have been a a way to actually talk to the agent directly. s225 00:23:08.240 --> 00:23:12.400 It could also have flagged what the agent was trying to send out. s226 00:23:12.400 --> 00:23:15.520 Now the last piece here is around agent sandboxing. s227 00:23:15.520 --> 00:23:25.919 So even if the instruction did land in that case there wouldn't have been a path to getting the credentials and no way to exfiltrate anything without setting off alarms. s228 00:23:25.919 --> 00:23:28.240 Now let's take a moment to zoom back out. s229 00:23:28.240 --> 00:23:31.919 Let's talk about these three incidents at the high level. s230 00:23:31.919 --> 00:23:41.440 We started off with the replet incident where we had an overprivileged agent that deleted production data despite an active code freeze. s231 00:23:41.440 --> 00:23:56.880 Then we had the co-pilot compromise which started with an ordinary email fishing scheme but gave an attacker access to an internal AI subscription which actively helped them find and excfiltrate private banking data that they were able to use s232 00:23:56.880 --> 00:23:59.760 to demand a ransom. s233 00:23:59.760 --> 00:24:12.960 Then we have the NX singularity attack which again exploited the supply chain turned developers coding tools against them hitting all three legs of the lethal trifecta at once. s234 00:24:12.960 --> 00:24:19.440 Something I'm going to be candid with you for a moment about is that aentic security is a very new field. s235 00:24:19.440 --> 00:24:27.200 Things are moving extremely fast and there is no single miracle fix that is going to protect you completely. s236 00:24:27.200 --> 00:24:33.360 If you have a vendor that's trying to sell you one, then that's worth being skeptical of. s237 00:24:33.360 --> 00:24:42.320 Even the lethal trifecta I've been talking about isn't a checklist to take off and if you pass it, then you're safe. s238 00:24:42.320 --> 00:24:50.559 It's more of a mental model for understanding how much risk your AI tooling is actually exposing you to. s239 00:24:52.240 --> 00:25:05.600 There are six key parts to a secure architecture and you're going to want to layer these six because even if an attacker or your own agent well-meaning but uh s240 00:25:05.600 --> 00:25:15.760 ultimately suggestable gets past just one then there's still five more to go and even the first three that I see here could have stopped every incident that I just showed you. s241 00:25:15.760 --> 00:25:19.279 Experimental separation is the key here, right? s242 00:25:19.279 --> 00:25:23.200 Agents should be executing in isolated remote workspaces. s243 00:25:23.200 --> 00:25:29.360 They should never be working on a developer's actual machine, never given a route to production. s244 00:25:29.360 --> 00:25:36.400 Like this is the highest leverage building block on this whole list because it doesn't matter what the prompt says. s245 00:25:36.400 --> 00:25:45.440 If there's no way to get to production, to get to the sensitive credentials, you're safe. s246 00:25:45.520 --> 00:25:49.039 This could have prevented the replet incident from happening. s247 00:25:49.039 --> 00:25:54.960 It could have shrked NX down to the blast radius of just one disposable container. s248 00:25:54.960 --> 00:26:03.440 Though I will say it wouldn't have protected from the copilot incident because copilot is living entirely inside of a SAS subscription. s249 00:26:03.440 --> 00:26:08.480 There's no way to run that in a separate execution environment. s250 00:26:08.480 --> 00:26:12.880 Next up is to stick to short-lived and scope credentials. s251 00:26:12.880 --> 00:26:17.840 We are not in the world where persistent broad scope tokens make sense. s252 00:26:17.840 --> 00:26:33.679 Having things like automatic expire expiration, narrow scoping, things that are issued per task instead of per issues that could have prevented replet which got its hands on a standing right access credential to make the deleting possible. s253 00:26:33.679 --> 00:26:43.760 as well as the co-pilot fishing where the attacker was able to keep on using the script the subscription even after the actual user session had ended. s254 00:26:43.760 --> 00:26:48.320 By the way, this is exactly where a product like coder comes in, right? s255 00:26:48.320 --> 00:26:53.200 What we do is we offer workspaces that are defined through code. s256 00:26:53.200 --> 00:26:56.480 They can spin up as isolated environments. s257 00:26:56.480 --> 00:27:10.159 They can be torn down easily and you're able to inject credentials inherently short-lived issued per task as opposed to living forever on someone's laptop. s258 00:27:10.159 --> 00:27:13.600 Next up is identity aligned access control. s259 00:27:13.600 --> 00:27:19.200 AI tooling should inherit the same identity provider enforcement as human access. s260 00:27:19.200 --> 00:27:24.400 Shouldn't be some parallel subscription with a door of its own. s261 00:27:24.400 --> 00:27:27.679 This is what could have directly stopped C-Pilot. s262 00:27:27.679 --> 00:27:40.480 A finance user's identity should never be able to grant an assistant permission to answer things like how to get around internal data loss prevention measures. s263 00:27:40.480 --> 00:27:46.799 Now that brings us to sort of the start of the agentic specific solutions. s264 00:27:46.799 --> 00:27:56.080 We have the model proxy layer which sits between the model and the agent to inspect, filter, and log every call that's going between them. s265 00:27:56.080 --> 00:28:04.640 This could have prevented the NX singularity incident by stripping the injected instructions out of external content before the agent read it. s266 00:28:04.640 --> 00:28:08.480 And it's also something that could have prevented the copilot incident. s267 00:28:08.480 --> 00:28:15.360 It could have flagged a request like, hey, how do I bypass DIP before it actually got answered? s268 00:28:15.360 --> 00:28:28.720 Agent sandboxing can be thought of as a firewall for agents restrictions on the file system and the network so that a misbehaving or compromised agent has a limited blast radius. s269 00:28:28.720 --> 00:28:40.399 This could have prevented the singularity attack because even in the case in which the instruction made it through to the agent, it could have stopped the agent from getting access to the credentials s270 00:28:40.399 --> 00:28:46.480 and prevented the agent from running the command to excfiltrate that data. s271 00:28:46.480 --> 00:28:49.840 Last up is having audit logging and observability. s272 00:28:49.840 --> 00:28:50.559 Right? s273 00:28:50.559 --> 00:28:59.279 We need to have every agentic action attributable to a human being should be logged somewhere outside of the agent's own reach. s274 00:28:59.279 --> 00:29:01.120 This could have prevented replet. s275 00:29:01.120 --> 00:29:06.880 This could have really helped out in catching the incident as it happened for co-pilot. s276 00:29:06.880 --> 00:29:17.520 Alerting on logs in real time as opposed to just buried in loads and loads of data would have meant catching these incidents in the matter of minutes instead of months. s277 00:29:17.520 --> 00:29:24.720 And again, all of these pieces are bits that coder offers an integrated solution for. s278 00:29:24.960 --> 00:29:32.240 To close off, the controls that could have stopped all three of these incidents are probably already inside of your organization. s279 00:29:32.240 --> 00:29:35.360 They're just waiting to be configured. s280 00:29:35.360 --> 00:29:38.559 Coder is something that can help with that. s281 00:29:38.559 --> 00:29:45.840 To end off, I have three things that I would recommend that you audit in this next week. s282 00:29:45.840 --> 00:29:52.559 One, I want you to track down every loan live token that agents in your organization can read. s283 00:29:52.559 --> 00:30:06.399 And two, I want you to figure out how many of your agents can find access to production, can find some way around your current limits. s284 00:30:06.399 --> 00:30:11.200 Three is I want you to check where your agents are actually executing. s285 00:30:11.200 --> 00:30:12.559 Are they sandboxed? s286 00:30:12.559 --> 00:30:14.640 Are they in managed workspaces? s287 00:30:14.640 --> 00:30:20.880 or are they just going wild on people's actual computers, actual laptops? s288 00:30:20.880 --> 00:30:24.960 One suggestion I have for you is to run a premortem with your team. s289 00:30:24.960 --> 00:30:29.600 Assume one of the three incidents that you just saw happens to you next month. s290 00:30:29.600 --> 00:30:37.919 Either ask people what they actually do or you can even pretend that it's actually happening and see what people actually do. s291 00:30:37.919 --> 00:30:43.120 That exercise is going to tell you exactly where your gaps are. s292 00:30:43.120 --> 00:30:45.679 So, that brings me to the end of my talk. s293 00:30:45.679 --> 00:30:56.240 Thank you all for watching, and hopefully you're walking away having learned a couple of things with some scary stories to tell at the next cocktail party you go to. s294 00:30:56.240 --> 00:31:00.399 If you would like to have these slides for reference, just shoot me an email. s295 00:31:00.399 --> 00:31:10.559 If you want to try Coder Workspaces or our AI governance solution, check out our repository for workspaces because it's free, open source, and self-hosted. s296 00:31:10.559 --> 00:31:15.840 you can just pull from our repository and get set up within 15 minutes. s297 00:31:15.840 --> 00:31:24.559 If you're interested in buying for AI governance or our other enterprise features, coder.com/cont is the way to go. s298 00:31:24.559 --> 00:31:27.679 Anyways, thank you for watching.