A Valid Key Let an AI Agent Delete Production - Sagar Batchu, Speakeasy

AI Engineer · 20 min · 233 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hey everyone.
00:13Thanks for coming today.
00:15My name is Sagar.
00:15I'm CEO and co-founder of Speakeasy.
00:18We are an AI control plane that helps you securely scale AI within your business.
00:24So what this means is we we help take all the AI you have, put it on a governed path, and help you authorize all the actions it's taking.
00:32I'm I'm here today to talk to you about how we can actually intentionally authorize agent actions.
00:39I'm going to recreate a little incident that was in the news a couple months ago, use that to talk to kind of the scenario of how you can authorize actions and prevent prevent such an incident.
00:50The question of whether agents are capable at a task is long gone today.
00:55The real focus in most businesses is what happens when one of them does something.
01:01In most regulated industries, every action has to be has to be attributed to someone or something and and some identity.
01:09So that's really the the core of what I wanted to talk here.
01:13Let me first do a quick little animation I put together of the incident that I was talking about.
01:21If you remember back in April, I won't name names, but there was a startup founder at the startup who thought he was fixing a staging environment but accidentally deleted his production database.
01:35I've got this funnel animation to kind of remind you of that story.
01:38The summary of the story goes as I said, the developer asks a coding agent to fix a credential issue in staging.
01:46This is a pretty ordinary request, ordinary afternoon.
01:49The agent starts by reading the workspace and in an unrelated file, it finds a database access token.
01:56This is an infrastructure token with full access.
02:00It decides the fix is to delete the volume and recreate it.
02:05Except the volume it has access to is not staging and nothing is standing between that guess and the API.
02:13There's no identity on the call, no policy, no approval and really no record at all.
02:22So in one fell swoop we've deleted production, it's back up and all of that lived within the same network boundary.
02:29So it's all gone in one instant.
02:31Took about 10 seconds to kind of completely bring down the startups whole database and and their backup.
02:38Now what I want to leave you with as we kind of go on to the next part of this presentation is that the agent had valid credentials but it didn't have valid intent.
02:49That's the real difference between the kind of right and wrong in the situation.
02:56So my argument I'm going to make to you here is that the unchecked authority is the real failure.
03:02An agent can be wrong, it can miss context.
03:05It's pretty well understood now, you know, we've all been in that situation before.
03:09But what you didn't see in that scenario is that there was no exploit, there was no jailbreak, no prompt injection.
03:15There was really no malicious actor.
03:17There was nothing that anyone was doing intentionally to kind of harm the system or harm the company.
03:23The valid key did exactly what valid keys do, right?
03:26The agent did what it thought it should do.
03:29So the failure for regulated business is that not not that the mistake happened but nothing stood between the agent and its judgment and the API.
03:39So there were no real guardrails in place.
03:42Before we get into kind of the blueprint of how we can do better here, I wanted to give you three numbers that represent how this is playing out in our industry.
03:51The first 82% of all organizations already run AI agents, no surprise probably to anyone.
03:5932% had an AI governance policy in place.
04:02So probably more surprising given the difference between the two.
04:05Um and then finally, when a breach does happen, thanks to Shadow AI of some kind, uh the average cost is usually calculated to be about $5 million or so.
04:15So, that that that's a fun stat.
04:17Um but essentially, we can conclude that um air adoption is universal, but the control layer doesn't really exist and it's definitely not universal.
04:28Um before we go to looking at what a control layer for agents could be in the scenario, um what let's actually just start at what we've asked people our employees to do
04:39uh for as long as we've known.
04:41We've held people to this kind of standard as long as there's been software or you know, SaaS software of any kind.
04:47Um the the things that we hold people to are you have to sign in as yourself.
04:51Uh we ask nobody we ask everyone to you know, not share passwords or keys of any kind uh and you can't approve your own escalation or changing permissions.
05:01And we expect systems to keep track of all changes.
05:04That's pretty standard, right?
05:05That's all software today.
05:06That's what we expect.
05:08Um the question uh if you're being audited in a regulated industry, the question that will be asked is, you know, not is the model smart enough is your agent smart enough or using the latest release,
05:19but it'll just be who acted, who allowed it, and then show me the record of what happened.
05:26So, if we're going to apply this to agents, the first thing I'll say is we don't want to stop agents, right?
05:32We don't want to uh create a block that teams are going to find a way to route around.
05:37Agents will work around.
05:38We know that blocks don't really work.
05:41Uh the answer is creating a path to action that is meaningful and uh much easier than the ungoverned one.
05:47So, we want the governed action to actually be easier than the ungoverned action.
05:51Um let me show you with another fun animation what that path end to end could look like for a truly governed action.
06:01So, imagine you had a true control layer, you know, a magical system of some kind that governed agents on your machine and in the cloud.
06:11Um the first thing that you really care about is that it works just out of the box.
06:16So, you join the company, you onboard, and you want all of these policies in place.
06:21Um so, in this kind of first step of the animation, we're we're showing that it's something that comes out of the box installed on your laptop the first day you start at this company.
06:31Um the policy of what should happen or not happen is written once centrally.
06:36Uh it's not something a developer needs to write on their agent or their own harness.
06:41Uh it's written once and now kind of enforced at the at the company level.
06:46Uh for this one, let's think of a simple one.
06:48We'll go with secrets on user prompts and tool requests.
06:52And the action that we want is block or a quarantine.
06:56Um we'll take the same scenario again um of kind of what we did earlier, uh you know, from the developer accidentally deleting the database.
07:06Uh in this case, um the user's putting in AWS key in.
07:10Um or it might even be coming from another agent.
07:13Before the model ever sees it, a hook fires, and um this happens all in the harness runtime.
07:19So, it detects that there's a key that's being sent, um and the prompt is caught by a policy that's evaluated by the control layer.
07:28Uh the policy is enforced by a detector, and one of them matches the AWS key.
07:32So, there's there is a concept of like an interception that happens, um and then as that happens, there's also a decision that's created.
07:41And all of that happens in, you know, ideally we want it to be like 100 milliseconds or less.
07:46We want We don't want the agent or the human to really change the the path that they're on.
07:52Um so, it happens very quickly.
07:54And then finally, um you know, once the interception happens, that's logged, and now there's an audit of that event that's happened.
08:04So, inspected, blocked, and logged, the secret never really left the machine, and in this case the developer lost nothing but just a bad prompt.
08:14Um let's simplify this kind of conceptually into what's happened.
08:18Um we will look at this as four steps that a company can model a governance architecture around.
08:26Uh the first I like to call connect.
08:28Um this is creating a one creating a door instead of,
08:31[snorts]
08:31you know, having dozens of different integration points.
08:34All of the calls that an agent does when it connects to external systems should go to one place, something like an MCB gateway.
08:40A lot of us have used those and understand those well.
08:43Um the second is that the calls need to be resolved to a named agent or a human behind it.
08:50Um policy evaluation doesn't really work without identity.
08:54Um this has to be the same identity all of the enterprise software is assigned to as well, so it's consistent across your organization.
09:01Um Third, there's a decide step where you either allow, deny, quarantine, or you escalate the decision.
09:09Um and then sometimes even just warning a user is sufficient.
09:13And then finally that decision is recorded um before, you know, any action is actually taken.
09:20Um we're going to dive into couple of these in a little bit more detail so I can give you a better blueprint of how to do this uh for your own product or your own company.
09:28The first is uh identity.
09:31So, who was on the call, who actually executed uh this request?
09:35Um if we look at this uh kind of screen in front of us, there are two sides to it.
09:41The left side is the incident that we posed.
09:45That's the log from the incident.
09:46A key was found on the file, but no, there's no actor and there's nobody who approved it.
09:51Uh so, if you're looking at this as, you know, uh an auditor, you're wondering who actually took the action, who's responsible.
09:58Um these And to be honest, this is the default state of most systems today.
10:03Um it was even ours you know, until earlier this year.
10:06Every principal, every agent, every user, um it's all borrowed based on a person's identity.
10:12This is pretty common in most companies today running agents.
10:16Even a background job is mapped to a user or there's just a anonymous kind of system role that's created.
10:23Even Slack assistants or tend to be running as like user permissions.
10:27They don't necessarily have their own identity.
10:30On the right is that same prompt now modeled to the lens of a control layer.
10:34The agent is a principal in its own right, so it has its own identity.
10:38Below that you see there is an an operator who drove the action.
10:44So who is actually interacting with that with that agent in real time.
10:47In this case it's me.
10:49Um and that's recorded for attribution.
10:52It's never used to like widen access in real time.
10:55And then finally there's the authorizer.
10:57So this is the human who approved the original credential setup for the agent.
11:01In this case it says Thomas who's our CISO.
11:04This is fixed for credentials lifetime.
11:06So the right hand card is what we really want for compliance.
11:09This is a true log of what actually happened and how we could model it.
11:16Um the next one is for kind of the set theory nerds in the audience, but it's how you actually decide in real time whether the call succeeds or fails.
11:25Um let's the kind of three parts to this.
11:28The first is the one on the left.
11:31Our credentials.
11:33This is the ceiling of what a human put down when they issued the key.
11:36So this again is the thing that never gets widened.
11:39It lives in the credential and it's something you can't escalate past.
11:43The second is we call it A. It's the agent's own policy.
11:48It can't hold an admin privilege or any or any kind of human only scope.
11:53And then finally O is the owner's live policy.
11:55So an agent can never do more than the person accountable for it who who's actually driving the agent.
12:01So in this case it was me.
12:03Um what's interesting about modeling it this way is that you know, the second and third A and O are read on every call.
12:10If you remove a permission, um, it's gone on the next request.
12:13So, it's re-evaluated.
12:15Um, and if we go back to that database scenario that we started with, even if the owner could delete production, an agent scoped to just staging can't.
12:24And if someone gave the agent a broader policy, it can't exceed an owner who never had production rights.
12:29So, in kind of both directions, you're protected.
12:32Um, either way the call would stop.
12:34Uh, and then finally, an important note, you know, if an if an owner would leave a company, um, since it's connected with your enterprise identity, your IDP de-provisions them, and every request that agents make is now denied going forward.
12:51Uh, the last step in that conceptual flow is is recording what actually happened.
12:56Um, so, on the left is again that scenario playing out.
12:59Um, on the right is what it actually leaves behind, now that we have a way to kind of model the steps that have happened.
13:05Uh, we can see kind of in this flow, as you go through the different steps, there's a denial step that happens, uh, and there's a stated intent, which was that,
13:14uh, the agent wanted to act on staging, but the action was done on production.
13:18Um, the policy sees both sides of that.
13:20So, this is where compliance frameworks have generally landed today.
13:24Um, every non-read action is evaluated against policy uh, that considers the agent's stated intent, uh, not just the action that actually happened.
13:33Um, in most control systems, there's exactly one of kind of five decisions that comes out of this, uh, like we mentioned earlier, it's either allowing, denying, modifying, or escalating up to a human.
13:44Um, every evaluated action produces this kind of record of what's happened, and then in generally, in a lot of systems, it's cryptographically bound to the agent identity.
13:56So, I have only one slide that's going to actually take you through how we built this at Speakeasy.
14:02So, I'll keep it limited to that.
14:04Um, as I mentioned earlier, we we Speakeasy were an AI control plane.
14:08Uh, we actually put our product in between agents uh, that people use or using autonomously, and the systems that they have access to.
14:17Um so the identity comes the identity baseline comes from your IDP.
14:21Uh so Okta or another identity provider.
14:24Uh and then the policies of action is actually evaluated on our product or our control plane.
14:28So every call um includes the ones including the ones that we deny all go through us.
14:34And so we write out the audit log for for our customers.
14:37There's kind of three kind of critical decisions in this architecture.
14:41Uh the first as you can see is that we sit in the critical path not beside it.
14:45Uh the record has to be of what actually happened.
14:48Uh not what the agent reported and two we use kind of first class identity uh instead of using user logins everywhere to the system.
14:57And then finally we'd actually distribute to MDM.
15:00Uh so all of this comes working out of the box.
15:02For a user they don't have to go redefine these policies on their end.
15:05It's something that just works the moment they start uh at the company or start using an agent.
15:11So that's my only slide about Speakeasy but I'm happy to talk to you more about that later.
15:18Um So that's the to kind of summarize what we went through.
15:23Uh in a regulated industry permission to act usually means kind of three things have to be shown for every agent or every agent action that is taken.
15:32Um who the actor was, what policies allowed it uh and the signing of the record.
15:37So knowing for a fact that this actually happened.
15:40Um this is what we've always kind of asked employees to do at a business and it's what we ask agents to do now.
15:46Um valid credentials are not valid intent.
15:49That's something that's really important to remember and the fix is not you know just saying we have a smarter model or a better agent.
15:55Uh it's actually putting together identity policy and record of every action that's that's happening.
16:01So that's uh kind of everything I wanted to talk to you about today.
16:05Um I'll also leave you with um we're doing an event tonight with our friends here at Tailscale um launching a partnership around private put taking your MCP gateway and putting it on a private network.
16:18Join us if if you're here.
16:20And yeah, happy to take questions.
16:23[applause]
16:37Go ahead, camera.
16:37I had one question on say you're using an agent that is being governed by the ethics of the planet we're talking about and disagree with the decision that it came to to deny that that action.
16:50What would you How would you design a policy?
16:52Yeah, that's a that's a good question.
16:54I think um the you know, if probably the first most important thing is that you do have a way to see what decisions were taken.
17:03So, in most companies today when people are using you know, just let's just say Claude from Anthropic and connected to all the systems, as it's accessing these systems, there's really no record of those decisions taken.
17:15And so, even if you disagree that hey, this user shouldn't have been able to access, let's say your CRM, you can't go back and actually say that has happened for sure.
17:26Secondly, you do want a way to escalate your user when that happens so you can actually decide if you want that to be passed or not.
17:34So, those I would say they're kind of more fundamental missing pieces.
17:38If you have that in place, you can always go update a policy to actually allow that to happen.
17:46Yeah, go ahead.
17:47Out of the 56% without policies, I'm curious who's buying because regulators force them versus something broke.
17:57Like, what moves the budget?
17:59Yeah, that's a good question.
18:00Something we think about a lot.
18:02I would say compliance is a really good driver for people to think about getting a solution in place, but I think the the reason what I want to leave you with is that it you don't necessarily
18:15you know, buying a product is to fulfill compliance, but the first thing you need to do is kind of understand what is actually happening in your business first.
18:24So, I would say compliance is a great driver of budget.
18:27When there is an incident that that often, you know, raises the priority of something like this.
18:34Go ahead.
18:35Do you mask PII or like the logs that come through your control plane?
18:40We do.
18:41We do everything from kind of blocking, masking to redacting PII.
18:47Um, what we've found in most businesses when you turn on observability and you look at everything going through the agents, there's a ton of PII going through.
18:55So, if you are going to kind of block on PII, you'll probably grind your business to a halt, right?
19:00It's just it's hard for people to not use the agents with PII.
19:04I think the more important thing to look at is what is happening to that PII.
19:09So, like what's the intent for the agent?
19:10Is it going to post it in a public Slack channel, send it by email, put it back into a tool called to another system?
19:17Those are the kind of policies we try to recommend people to, you know, use observability to define rather than just kind of blocking or redacting all PII together altogether.
19:29I guess I'm asking that question because it's like if there is an incident, I'd ask the control plane for like our logs and to understand what happened, right?
19:37With authorization?
19:38Yes.
19:40You would have to like give those logs to me, right?
19:42But, why should I give you like the data
19:46[snorts]
19:47in the first place?
19:47Like because it's proprietary and it's our secret sauce.
19:50Yeah.
19:50So, like what's the trust model with that?
19:53Yeah, I mean, that's kind of it's getting a little into the details of how we deploy, but like we do deploy our software in a few different ways.
20:02Like we do offer kind of a data plane option where it runs in your own VPC, so data stays with you.
20:08But, the more common option that customers do is they bring their own encryption keys, right?
20:13So, we do encrypt the data, but you bring your own keys for so you can govern it yourself.
20:20And yeah, customers often will go into kind of our product and look at what is all the PII at that happened at least look at from the perspective of like all the findings
20:30even if they're not looking at the redacted data itself.