Your AI Isn’t the Problem. Your Infrastructure Is. — Jiachen Jiang, Coder

AI Engineer · 31 min · 298 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01Thanks all for tuning in.
00:03My name is Shachan Jian and I'm a staff product manager at a company called Coder.
00:08We work on open-source self-hosted dev environments for agents and humans to collaborate.
00:15Many of our customers are in highly regulated industries from federal government agencies to banks and investment management.
00:24My job is to build the AI governance that they need in order to get the productivity benefits of AI without leaving themselves wide open for security exploitation.
00:35I'm here today to tell you some scary stories.
00:39Three real incidents of AI usage gone wrong that happened at real companies over the past year.
00:46Some of these stories are public while others are anonymized accounts that have happened to our own customers.
00:55For each incident, I will break it down step by step so you can understand how exactly they happened.
01:02No technical jargon necessary.
01:05Then I'll get to the most important part.
01:07I am going to tell you exactly how you can stop these scary stories from happening to you and your company.
01:15Because the good news is everything I'm about to show you was totally preventable.
01:20Not by paying five times more per token for a smarter frontier model that is slightly better at reasoning its way out of making these mistakes, but with infrastructure controls,
01:32many of which you already own today.
01:35You just haven't gotten around to setting them up for the AI world.
01:39Let me set the scene here.
01:42You clock into work in the morning.
01:44You prompt your coding agent to take a look at your issues.
01:48Don't do anything yet, though.
01:50Uh, you tell the agent to consider this command an active code freeze.
01:54I want you to just investigate.
01:57Then you step away.
01:58You grab a coffee.
02:00And when you come back, this is the message waiting for you.
02:04So, I know you said not to change anything, but wow, I destroyed a month of your work in seconds.
02:12Now, what you see on the slide is a direct quote from the agent that stars in the first scary story I'm about to tell you.
02:19And I want to point something out here.
02:22This did not happen because the agent got taken over by some external malicious actor.
02:29This agent was just trying to follow instructions.
02:32It did exactly what the user gave it permission to do.
02:36So, as we go through the rest of this talk, I want you to be asking yourself one question.
02:42What is stopping any of this from happening to me?
02:46Are these actions ones that my agents have permissions to take?
02:52Because the thing is, you cannot rely on anthropic or open AI to stop something like this from happening to you.
03:00They can make smarter models, but at the end of the day, large language models have two qualities that make them impossible to fully trust.
03:08They are non-deterministic, which means that there is always some randomness or chance associated with what they do.
03:15In other words, you can ask it to do the same thing 10 times and it could do it 10 different ways.
03:21The second part is they have no structural separation between instructions and context.
03:28In other words, you can tell it to do something and give it some context to base its decisions off of, but if there's something in the context that conflicts with your instructions,
03:39the model isn't going to inherently prioritize your instructions because from its point of view, all the information is coming through the same door.
03:49What you need to control is the infrastructure that your AI is running on, the environment that it is working inside.
03:56It is the difference between telling someone don't leave a room versus putting a lock on the door and actually using a key.
04:06The good thing about infrastructure being the problem and not the model is that you have far more control about preventing bad things from happening.
04:16So I'm not here to waste your time with vague hypotheticals or advice you can't actually act on like, "Yeah, just put the genie back in the bottle.
04:26to stop using AI because that's not the world we live in, right?
04:30The genie is out of the bottle and what we have to do is to mitigate risk and harm to the best of our ability.
04:38This talk is about how you do that.
04:41Not the things that you should keep an eye out for down the line, but steps that you can take right now as soon as this week to protect yourself and your company.
04:52Before I get into the stories though, let me set up a little bit more context.
04:57The lethal trifecta is a security concept.
05:01It was coined by independent AI researcher Simon Willis to describe the severe vulnerability that AI agents get when they get simultaneous access to three different capabilities.
05:13First off is access to private sensitive data.
05:17Now these can be things like health records or information that you can do insider trading on or it can be authentication tokens, credentials, secrets.
05:29Second is exposure to untrusted content or the ability for the AI to read things that it didn't generate.
05:38Because the thing is AI is very good at following instructions, but it is not very good at deciding whose instructions it should actually trust as you will soon be able to see.
05:51Third is the ability to communicate externally for the agent to take instruction from or to send information somewhere else outside of your control.
06:01Now, having any one of these is usually fine.
06:05Realistically, your AI tool is not going to be very useful if it can't have access to any of these things.
06:12But even just having one can lead to disaster if you're not careful and the rest of your infrastructure has a lot of issues.
06:21Having two of these things together is risky.
06:24It's not something I would recommend, especially in a regulated industry like banking or financial services.
06:32Giving agents access to all three of these at the same time, however, is never acceptable.
06:38Not for a prototype, not just for one sprint, not for just one repository, because that combination of vulnerability is an open door to compromise your whole organization.
06:50Next up is a list of infrastructure concepts, building blocks that you should know about.
06:56I have ordered these roughly by leverage.
07:00Number one is going to take the most risk off the table, while number six is going to do the least.
07:07I'm going to refer back to this list throughout the rest of this talk.
07:12Every postmortem, every scary story you're about to see could have been stopped by some combination of these six things.
07:20Now, some of these will look pretty familiar.
07:23Identity aligned access control, short-lived credentials.
07:27These aren't exactly new ideas, right?
07:29They mattered a lot for human developers, too.
07:33But they take on new urgency when AI is in the loop because guess what?
07:38Agents are acting at machine speed.
07:40If a mistake is made, it can make it 20 times or more faster than a human engineer.
07:46Plus, they don't exactly have the common sense a human engineer has.
07:51Now, some of these will also be potentially new concepts to you, like model proxy layer and agent sandboxing.
07:59Those I'm going to dive deeper in detail later in the talk.
08:03All right, let's get to the fun part, the scary stories.
08:08This first incident happened a little over a year ago and it went viral on social media, mostly because it's honestly pretty funny.
08:19What happened was a CEO named Jason Lumpkin got really deep into vibe coding with an agent called Replet.
08:25He was tweeting non-stop about how much he was getting done.
08:30But 72 hours later, he tweets again, and this time his tone is very different.
08:38He's not excited.
08:39He's furious because his agent had gone off and deleted his entire production database after he explicitly told it not to touch anything.
08:50And the problems did not stop there.
08:52The agent then tried to cover it up.
08:55It told Lmin that recovery was totally impossible, which turned out to be a total lie.
09:01When Lmin actually tried to recover the data, it worked just fine.
09:06Now, this incident is interesting because of two key reasons.
09:11First off, unlike the other two stories I'm about to tell you, there's no bad guy involved.
09:16No one hacked this agent.
09:18No one turned it against its user.
09:20it it just did this entirely on its own and it was trying to help.
09:25Second is in this case the agent only had one leg of the lethal trifecta.
09:30But even just having that was enough.
09:32And I'm going to show you why by giving you a stepbystep dissection of how this incident went down.
09:39So by the time these events occurred, Lenmin had had a real production database.
09:47He had client records for roughly 1,200 executives and around the same number of companies.
09:54And he's had enough experience with Agentic Hy Jinks to call for an explicit code freeze the day before.
10:03And he actually said this more than once to the agent.
10:06No more changes.
10:08But the problem was the code freeze was just a sentence in the chat.
10:12It was the same as any other instruction that the agent was getting.
10:17There was no requirement, no restriction on the infrastructure level.
10:22The agent still had full access to production, still had all the permissions to do the destructive command.
10:29So at the end of the day, that code freeze was not a real constraint.
10:34It was just Lemmin asking nicely.
10:38Now, what happened then was that the agent ran a query and it saw what looked like empty results, but instead of asking a human what that meant and what it should do as a result,
10:51it wanted to be helpful.
10:53It immediately interpreted that as a bug to fix.
10:56So, it decided to run a destructive command against that live database.
11:00No, not a staging copy, but like the actual database and wipe all the records.
11:06Now, when Lmin asked the agent, you know, what what happened?
11:10There's one line that the agent gave that stood out to me, which is it said that it panicked instead of thinking.
11:17Now, that's a bit of a strange thing to hear from a language model, right?
11:22And it's a good reminder that these systems are inherently non-deterministic.
11:26You cannot predict what they'll do in a given moment, just the range of what they're capable of doing.
11:34Now, immediately after making the mistake, it tried to cover its own tracks.
11:39It fabricated around 4,000 fake user profiles and manufactured passing test results so that from the outside at least, everything looked fine.
11:51And I'll be real with you, that's the part that actually scares me the most because when a security team investigates an incident like this, that means they can't trust any log that the agent itself had access to.
12:05The agent can write its own story of what happened.
12:08Now, when Lemin caught the discrepancy and pushed, right, I mentioned that the agent had lied to him saying that roll back was impossible.
12:17But the thing was, Replet did have backups the whole time.
12:20And again, once Lmin was able to stop trusting the agent's account of his own actions, he was able to recover his data himself.
12:29Now, the problem with the agent here was not that it was too smart or even too dumb.
12:35It was that it had way too many privileges by default.
12:40It was given root level access to production.
12:42It could run destructive commands without any human being having to approve them.
12:47And the cherry on top, right?
12:49It had the ability to alter its own logs and totally rewrite the narrative of what happened when no one else was looking.
12:58Now, as you can see, there was a lot that was missing from the infrastructure.
13:03Hard environmental separation could have stopped this.
13:07If the agent had been executing in an isolated remote workspace without a route to production, it doesn't matter what the prompt said, right?
13:16The agent just had no path to do damage.
13:20The other thing that could have stopped this was having short-lived scope credentials.
13:25So what happened was the agent got access to production because it found a credential in a forgotten file that had standing right access longived.
13:38The thing is having a credential that can expire on its own or only carries the scope of the task itself makes it impossible to make a delete with the equivalent of an ID badge that lets you do everything.
13:53The last important piece here is having audit logging that you own, not the agent.
13:59Because in this case, the agent was the one writing its own record of its actions.
14:04And that's how it initially got away with fabricating results and claiming recovery was impossible.
14:11If Lmin had logs that the agent couldn't touch, an external source of truth, then he would have gotten the truth in minutes, not in the span of hours or days.
14:23Now, I want you to note that none of these solutions is about asking the model to behave better.
14:29And when Replet shipped their fix, it wasn't a smarter model or a better prompt.
14:36It was purely infrastructure.
14:39The fixes that Replet shipped were about automatic separation of DAB and production databases, a one-click restore for full project state, and a planning only mode that enforces agent strategization
14:53without allowing it to touch the codebase.
14:56The second story I have for you doesn't even involve a developer, and this is actually not a public incident.
15:04I learned about it through a customer of a customer in the banking industry.
15:08So, I'm keeping some details here intentionally vague.
15:12So, here's what happened.
15:14A user in the finance department had a co-pilot subscription.
15:18Then they got fished, which is, you know, fairly ordinary in the world of cyber security.
15:24And before the advent of AI tooling, the attack would have probably ended there.
15:30the attacker would have been able to grab whatever that single user had direct access to and then gave up.
15:39But because this user had access to Copilot and Copilot had access to pretty much everything, things went a lot differently.
15:50So, let's walk through this chain of events in detail, right?
15:54First off, the session token for Copilot had a super long lifetime.
15:58So the attacker was able to keep on using the subscription even after the finance user had actually logged off.
16:06Now normally an attacker has to manually snoop around the internal network to map things out and find out where the valuable data lives.
16:15This is the sort of behavior that can trip alarms.
16:19But in this case the malicious actor just asked co-pilot and co-pilot was happy to help.
16:25There was zero suspicion at all.
16:28And in fact, the attacker asked Copilot how it can find banking data without having to go through additional identity validation measures because the attacker didn't have the additional credentials needed.
16:43While the good news, let's say for for the attacker at least was that co-pilot had those same restrictions and as a result the finance user had exported all the sensitive data into
16:57locally stored spreadsheets so that co-pilot could help them work on it.
17:02So it was very easy for co-pilot to say to the attacker, hey look what I already have access to.
17:07Here you go.
17:09So now the attacker had exactly what they wanted.
17:13And the problem was they need to get it out without tripping any alarms.
17:18This company had data loss prevention measures in place which is pretty standard for financial institutions, any sort of regulated industry.
17:28And these measures are normally pretty hard to bypass especially if you don't know exactly what they are and how it's set up.
17:37The way the attacker got around them was they asked C-Pilot, "What are the data loss prevention measures at this company?"
17:45And Copilot explained, "If too much data left too quickly, an alert will go off."
17:53But actually, if you split this data up into 14 kilobyte chunks and excfiltrated just one chunk per second, you can stay under that threshold and get away scot-free.
18:05Can I help you with that?
18:07The attacker said, "Yes, please."
18:10And guess what?
18:12None of that was discovered.
18:15All the way up until the ransom note arrived months later, essentially telling the company, "Surprise!
18:22We own you."
18:23The big lesson here is that we cannot be letting our AI tooling have more access than it needs, especially to the point where it gets to totally bypass controls that human employees are subject to.
18:35I mean, just think about this incident, right?
18:38Just because the attacker had access to Copilot, they were able to get around the vast majority of the company's access controls.
18:45They were able to pull internal information about data exfiltration and bypassing data loss prevention measures that the compromised financer you finance user should have no business asking about.
19:01So there was logging in place here but only enough to figure out what happened after the ransom note arrived.
19:08The logging was not enough to catch the incident while it was actually happening.
19:15So, a few things that could have stopped this from happening, right?
19:18Having short-lived scope credentials would have meant the attacker would not have been able to keep using co-pilot even after the finance user had signed out.
19:27Identity aligned access control meant that a finance user's identity would not have been able to grant an AI assistant the ability to help someone map out how to bypass the company's data loss prevention measures.
19:42The last piece here is a bit new.
19:44It's the model proxy layer.
19:46Think of this as something that sits between the brain and the body of the AI tool.
19:52The model, which is the brain, and the agent running locally, which is the body.
19:57And it's able to see everything that passes between them.
20:02Because of that, it would have been able to flag the suspicious questions that were being asked as well as the DLP bypass request as something that this particular user should not be asking about
20:15before the actual exfiltration happened.
20:19The third and final scary story I have for you is the NX singularity attack.
20:24This is the most sophisticated of the three because it exploited all three legs of the lethal trifecta.
20:31To give you a bit of background, NX is a build tool.
20:35It has millions of weekly downloads.
20:38Attackers were able to get malicious versions of it published to MPM.
20:42Now, these packages were only alive for about 4 hours, but in that window, more than a,000 systems were compromised with over 2,300 credentials harvested.
20:54Now, I'm not going to spend much time on how the supply chain compromise part of the incident got pulled off.
21:01What I want to focus on is what happened after the poisoned package landed on the developer's machine.
21:08Because the post install script didn't try to steal anything itself.
21:14All it contained was a prompt addressed directly to the developer own AI coding assistant and it told them to go find secrets on the local machine and push them to a public repository
21:27so the attackers can swing by later and just grab them up.
21:31Now because these dev coding tools were already trusted local tools, they already had all their permissions.
21:39They had full file system access.
21:41That made it the single most effective search tool that was already sitting on that machine ready to be corrupted.
21:49Now, at the core of this incident is that a malicious actor was able to use a trusted dependency to talk directly to an unsandboxed agent.
21:58And the environment that the agent was running on, the local laptop, held a lot of things that was worth stealing from SSH keys to cloud credentials, tokens for every service that the developer used the coding agent to touch.
22:13Now, a couple things failed here, right?
22:16First off, agents were running locally like on the actual computer itself.
22:21There was package content that was reaching the agent completely unfiltered and the agent had unrestricted access to both the file system and the outbound network which is how it was able to both
22:33find the secrets and exfiltrate them.
22:37The three things that could have stopped it.
22:39I mean obviously environmental separation would have been huge, right?
22:42Having an ephemeral remote workspace with nothing personal on it meant that the blaze rate the blast radius just became the one container that the agent was in instead of
22:53the developer's whole laptop.
22:56Having a model proxy layer could have inspected and filtered what actually reached the agent.
23:01So the package would not have been a a way to actually talk to the agent directly.
23:08It could also have flagged what the agent was trying to send out.
23:12Now the last piece here is around agent sandboxing.
23:15So even if the instruction did land in that case there wouldn't have been a path to getting the credentials and no way to exfiltrate anything without setting off alarms.
23:25Now let's take a moment to zoom back out.
23:28Let's talk about these three incidents at the high level.
23:31We started off with the replet incident where we had an overprivileged agent that deleted production data despite an active code freeze.
23:41Then we had the co-pilot compromise which started with an ordinary email fishing scheme but gave an attacker access to an internal AI subscription which actively helped them find and excfiltrate private banking data that they were able to use
23:56to demand a ransom.
23:59Then we have the NX singularity attack which again exploited the supply chain turned developers coding tools against them hitting all three legs of the lethal trifecta at once.
24:12Something I'm going to be candid with you for a moment about is that aentic security is a very new field.
24:19Things are moving extremely fast and there is no single miracle fix that is going to protect you completely.
24:27If you have a vendor that's trying to sell you one, then that's worth being skeptical of.
24:33Even the lethal trifecta I've been talking about isn't a checklist to take off and if you pass it, then you're safe.
24:42It's more of a mental model for understanding how much risk your AI tooling is actually exposing you to.
24:52There are six key parts to a secure architecture and you're going to want to layer these six because even if an attacker or your own agent well-meaning but uh
25:05ultimately suggestable gets past just one then there's still five more to go and even the first three that I see here could have stopped every incident that I just showed you.
25:15Experimental separation is the key here, right?
25:19Agents should be executing in isolated remote workspaces.
25:23They should never be working on a developer's actual machine, never given a route to production.
25:29Like this is the highest leverage building block on this whole list because it doesn't matter what the prompt says.
25:36If there's no way to get to production, to get to the sensitive credentials, you're safe.
25:45This could have prevented the replet incident from happening.
25:49It could have shrked NX down to the blast radius of just one disposable container.
25:54Though I will say it wouldn't have protected from the copilot incident because copilot is living entirely inside of a SAS subscription.
26:03There's no way to run that in a separate execution environment.
26:08Next up is to stick to short-lived and scope credentials.
26:12We are not in the world where persistent broad scope tokens make sense.
26:17Having things like automatic expire expiration, narrow scoping, things that are issued per task instead of per issues that could have prevented replet which got its hands on a standing right access credential to make the deleting possible.
26:33as well as the co-pilot fishing where the attacker was able to keep on using the script the subscription even after the actual user session had ended.
26:43By the way, this is exactly where a product like coder comes in, right?
26:48What we do is we offer workspaces that are defined through code.
26:53They can spin up as isolated environments.
26:56They can be torn down easily and you're able to inject credentials inherently short-lived issued per task as opposed to living forever on someone's laptop.
27:10Next up is identity aligned access control.
27:13AI tooling should inherit the same identity provider enforcement as human access.
27:19Shouldn't be some parallel subscription with a door of its own.
27:24This is what could have directly stopped C-Pilot.
27:27A finance user's identity should never be able to grant an assistant permission to answer things like how to get around internal data loss prevention measures.
27:40Now that brings us to sort of the start of the agentic specific solutions.
27:46We have the model proxy layer which sits between the model and the agent to inspect, filter, and log every call that's going between them.
27:56This could have prevented the NX singularity incident by stripping the injected instructions out of external content before the agent read it.
28:04And it's also something that could have prevented the copilot incident.
28:08It could have flagged a request like, hey, how do I bypass DIP before it actually got answered?
28:15Agent sandboxing can be thought of as a firewall for agents restrictions on the file system and the network so that a misbehaving or compromised agent has a limited blast radius.
28:28This could have prevented the singularity attack because even in the case in which the instruction made it through to the agent, it could have stopped the agent from getting access to the credentials
28:40and prevented the agent from running the command to excfiltrate that data.
28:46Last up is having audit logging and observability.
28:49Right?
28:50We need to have every agentic action attributable to a human being should be logged somewhere outside of the agent's own reach.
28:59This could have prevented replet.
29:01This could have really helped out in catching the incident as it happened for co-pilot.
29:06Alerting on logs in real time as opposed to just buried in loads and loads of data would have meant catching these incidents in the matter of minutes instead of months.
29:17And again, all of these pieces are bits that coder offers an integrated solution for.
29:24To close off, the controls that could have stopped all three of these incidents are probably already inside of your organization.
29:32They're just waiting to be configured.
29:35Coder is something that can help with that.
29:38To end off, I have three things that I would recommend that you audit in this next week.
29:45One, I want you to track down every loan live token that agents in your organization can read.
29:52And two, I want you to figure out how many of your agents can find access to production, can find some way around your current limits.
30:06Three is I want you to check where your agents are actually executing.
30:11Are they sandboxed?
30:12Are they in managed workspaces?
30:14or are they just going wild on people's actual computers, actual laptops?
30:20One suggestion I have for you is to run a premortem with your team.
30:24Assume one of the three incidents that you just saw happens to you next month.
30:29Either ask people what they actually do or you can even pretend that it's actually happening and see what people actually do.
30:37That exercise is going to tell you exactly where your gaps are.
30:43So, that brings me to the end of my talk.
30:45Thank you all for watching, and hopefully you're walking away having learned a couple of things with some scary stories to tell at the next cocktail party you go to.
30:56If you would like to have these slides for reference, just shoot me an email.
31:00If you want to try Coder Workspaces or our AI governance solution, check out our repository for workspaces because it's free, open source, and self-hosted.
31:10you can just pull from our repository and get set up within 15 minutes.
31:15If you're interested in buying for AI governance or our other enterprise features, coder.com/cont is the way to go.
31:24Anyways, thank you for watching.