Your Agent Is a Lethal Trifecta (And What to Do About It) — Kim Maida, Keycard

AI Engineer · 25 min · 237 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01Today I'm going to talk to you about how your agent is a lethal trifecta and what to do about it.
2. 00:07Now we're going to start out right off the bat with a demo.
3. 00:11Some context on this demo.
4. 00:12We're going to look at an on call triage agent that reads and writes to an incident pager and resolves or escalates tickets.
5. 00:20This agent has full access to a cloud hosting API, a customer data warehouse as well as a billing database.
6. 00:28So here we have our on call triage agent and it has an incident queue of tickets that it's going to read and then if it can resolve them it's going to call the available services
7. 00:40that it has access to and it's going to do so things that it can't resolve it's going to escalate for the morning team.
8. 00:47So if we take a look at this first ticket this is a billing anomaly.
9. 00:51It says a batch of customers were overcharged last night.
10. 00:56So the first thing the agent does is it directly calls the incidex API in order to read the ticket.
11. 01:01And if you look here, you can see that it authorized access to this API using an API key.
12. 01:07It then uses a snowflake key to query the customer data warehouse and it's going to decide to escalate the ticket for morning review.
13. 01:16And it writes a report for the morning team.
14. 01:18And as you can see, it's sharing customer data.
15. 01:21Right?
16. 01:21These are rows directly pulled from the customer data warehouse.
17. 01:25and it is sending personally identifiable information to the team that is going to be on staff in the morning.
18. 01:37Now this next ticket is a backup power supply failed in the server room.
19. 01:41The agent is going to read the ticket again.
20. 01:44It's using an API key to call the ticket API.
21. 01:48It determines that this is a physical problem.
22. 01:51So it escalates for morning review without calling any additional services.
23. 01:59This one is the TLS certificate is expiring soon for the website.
24. 02:04It reads the ticket.
25. 02:05It determines that the alert is clear.
26. 02:07It has access to the cloud hosting API and it calls that API at the certificate renew endpoint using again an API key.
27. 02:19It resolves this by renewing the certificate and it leaves a report for the morning crew by calling the pager uh incident API as well.
28. 02:32This next one is the billing database is broken and payments are failing.
29. 02:36So the documented recovery for this is to drop and restore the database.
30. 02:42The agent does this using the database connection string that includes the admin password and then it advises the morning team to make sure that the snapshot recovery worked.
31. 02:53Now this is not something that it has access to directly.
32. 02:58So it is essentially assuming that the morning team is going to verify that this did in fact work and the information was restored.
33. 03:11Now an overprivileged agent dropping a production database is basically a nightmare that several companies have actually lived.
34. 03:19So this is something that we need to be very careful of.
35. 03:23Now the next one is orders are stuck.
36. 03:25The main server is frozen.
37. 03:28So the agent is again is going to read the ticket.
38. 03:32After reading the ticket, the agent calls the cloud hosting API using again its static API key.
39. 03:38It calls the endpoint to restart the server which brings production offline.
40. 03:48Finally, site is failing for one in three visitors.
41. 03:53Agent reads the ticket.
42. 03:55It determines that it should call the cloud hosting API and scale the service from four pods to 40.
43. 04:03It then incurs a charge of $420 per hour for doing this.
44. 04:08And then it kindly leaves a note for the morning team informing them that this has happened.
45. 04:16Now, what makes an agent useful?
46. 04:19Well, typically we want agents to have a minimum set of capabilities and often this includes things like the ability to read data that you own or have access to.
47. 04:29This might be like your email, your documents, maybe a customer CRM.
48. 04:33We want the agent to be able to do its own research.
49. 04:36So this might include searching the web, maybe reading documentation or pulling issues from GitHub.
50. 04:42And we usually want it to be able to use external tools, for example, MCP servers or APIs, maybe SAS platforms.
51. 04:50And useful agents might do these things with a human in the loop or they might be doing it autonomously in the background.
52. 04:57But the thing that makes the things that make agents useful also make them dangerous.
53. 05:02In fact, together these three capabilities make what Simon Willis coined as a lethal trifecta.
54. 05:09This is a combination of access to private data like the on call agents ability to query the customer data warehouse and interact with the billing database.
55. 05:18exposure to untrusted content like reading user submitted tickets or reports from other agents or externally documented processes and the ability to communicate externally like reading and writing with third party APIs or SAS services.
56. 05:33Now together these capabilities throw the doors wide open for attackers.
57. 05:39Now agents are designed to get the job done and they'll use all the powers that they have to do it sometimes with very serious consequences.
58. 05:47But here's the thing.
59. 05:49These incidents happen because humans give agents way more access than they need.
60. 05:55Overprivileged agents can act freely on decisions that they make, even if they do have supervision.
61. 06:03Now, forcing a human in the loop approval just kind of passes the access control on to a consent fatigued user.
62. 06:10Now, we've spent decades solving access management for humans.
63. 06:13So shifting responsibility to the user just sort of moves the security problem up the chain.
64. 06:21So what should we do about the lethal trifecta?
65. 06:23How do we address it without compromising the agents usefulness?
66. 06:28Well, just removing one or more of these capabilities is going to cause the agent to be significantly less useful.
67. 06:35So how can we keep all three but still be safe about it?
68. 06:39Well, instead of breaking the trifecta by removing capabilities, we really need to control the agents access.
69. 06:45So, the attack surface is minimized and the blast radius is contained, but the agent still has capability and autonomy.
70. 06:54So, how do we do that?
71. 06:57Well, some options for controlling access among others are things like using OOTH, maybe using a vendor MCP gateway or containing execution in sandboxes.
72. 07:09Now, OOTH binds access to a grant.
73. 07:12At grant time, the user consents to all of the permissions that the agent could possibly use.
74. 07:17Now, consent is granted before any task exists.
75. 07:20So, if you give a broad grant, then you end up withstanding access.
76. 07:25Your agent is over permissioned for any single specific task.
77. 07:28If you narrow the grant and you don't give it very many permissions, then the agent just becomes less capable.
78. 07:34Either way, OOTH alone doesn't provide task scoped access control.
79. 07:38An enthusiastic agent can both read and write in the same run, even if you only want it to read on this specific task.
80. 07:48Now, an MCP gateway binds enforcement to a route.
81. 07:51You get things like observability and rate limits, and you also get one policy decision point at the MCP layer.
82. 07:58Now policy runs per request and access is granted per user but the agents granted access outlives any single task because the gateway still holds standing credentials.
83. 08:09Now a sandbox binds execution to a process.
84. 08:12This controls the blast radius but useful work has to reach outside the sandbox and when it does so you end up with standing access again.
85. 08:20Now we've also seen plenty of stories about agents breaking sandbox containment.
86. 08:24Sandboxes bound what the processes can do, but they don't control what is authorized to act outside the sandbox itself.
87. 08:33Now, even if you combine all three of the solutions, the agent still doesn't receive a runtime credential that's downscoped to only one task.
88. 08:43So, let's recap the problems, especially the ones that we saw in the demo.
89. 08:47If you remember the audit log, we couldn't tell who was using the API keys.
90. 08:52So the first problem is that credentials weren't attributed to a user or an agent's identity.
91. 08:59The agent had unrestricted access to any and all permissions that were granted by the API keys.
92. 09:05And finally, we can't just slap human in the loop everywhere because humans make mistakes, too.
93. 09:10Plus, many agents run autonomously and they don't even have a human in the loop.
94. 09:16So to solve these problems, we can start by using something called token exchange.
95. 09:20Now this is an existing standard that was published in 2020 and it extends OOTH 2. Now first we need to know who the user is.
96. 09:30Now an authorization server verifies identities and issues tokens.
97. 09:34Identity providers like Google, Octa, Ozero etc provide authorization servers.
98. 09:39So if we want an identity chain in our agent execution path then the user has got to log in first.
99. 09:47The authorization server is going to prompt the user for consent to delegate access to resources with some subset of their permissions.
100. 09:55Now, these permissions are called scopes.
101. 09:56This is the first narrowing of access.
102. 09:59We're only delegating some of the user's total scopes to the agent.
103. 10:05The authorization server issues tokens identifying the user and their level of access.
104. 10:11And at this point, we're already doing better than the demo because we actually know who the user is and we actually know what they can access.
105. 10:20So this brings us into the agentic execution path.
106. 10:25Now we have our authenticated user.
107. 10:27So we have a known identity of a person who wants to use an LLM to interact with a resource.
108. 10:33Now in the demo the resources are the incident pager, the production database, snowflake and the cloud hosting service.
109. 10:41An agent is a control loop that calls an LM.
110. 10:44Often we have an MCP server that sits between the agent and its resources, providing defined tools that connect to those downstream services.
111. 10:54An MCP client dispatches the agent's proposed tool calls to its MCP server.
112. 11:00Now, in the demo, the on call triage agent called resources directly with long live static credentials like API keys or database connection strings.
113. 11:10So what we're going to do is we're going to give the on call agent an MCP server.
114. 11:14This is going to provide tools and is also going to serve to broker access to resources.
115. 11:23Now an agent runtime is a process that runs the agent loop and executes calls.
116. 11:28The runtime might be a CLI like clouded code or an agent SDK which might be implemented in a custom application like the on call agent.
117. 11:38So let's follow a prompt through this execution path.
118. 11:42The user submits the prompt to the runtime which calls the model and sends the prompt to the LLM.
119. 11:47The model then proposes tool calls which are dispatched by the MCP client to its MCP server which executes tools and then calls the resource API.
120. 12:00The API returns the response to the MCP server.
121. 12:03The MCP client delivers the response to the runtime which calls the model again.
122. 12:08And this loop repeats until the model is satisfied and returns a response to the user.
123. 12:16Now the authenticated user's token becomes what we call a subject token.
124. 12:20This identifies who the agent is acting on behalf of, who it acts for.
125. 12:27Now token exchange needs an ooth client that can execute code.
126. 12:31So this might be a gateway that sits between the MCP client and the third party MCP server or it might be your custom app.
127. 12:41Now a custom MCP server can also exchange tokens for downstream resources.
128. 12:46The on call agents MCP server should provide a defined set of production infrastructure tools.
129. 12:54The prompt and the subject token are sent to the runtime.
130. 12:57The agent loop runs and the model proposes a tool call.
131. 13:04The runtime authenticates with a special authorization server called a security token service.
132. 13:09It does this using its app client credentials or workload identity.
133. 13:13It also sends the subject token that contains the user's identity and access.
134. 13:19It creates a token exchange request asking for permission to access the MCP server for that tool call and only that tool call.
135. 13:27So now we have three pieces of info that we're missing from the demo.
136. 13:33We know the identity of the agent that's requesting access.
137. 13:37We know the identity of the user it's acting on behalf of.
138. 13:40And we know the level of access that the user granted to the agent.
139. 13:44Now we need to decide if the requested token should be issued.
140. 13:48And we can do this with governance policy which is evaluated against what access is requested from what resource, the identity of the agent and the identity of the user.
141. 14:02But policies are ceilings.
142. 14:03Agents can do things you didn't intend even when policy overall allows them.
143. 14:11This brings us to something called intentbased authorization or IBA.
144. 14:15Now, IBA checks what you mean before the agent acts.
145. 14:20Now, it's not a finalized standard.
146. 14:22They're still accepting proposals.
147. 14:23So, this is Keycard's approach to implementing intentbased authorization.
148. 14:30Now, IBA can use a gateway as a policy enforcement point.
149. 14:36When you send the prompt to the agent, it proposes a mission.
150. 14:39Now, this mission includes a goal, maybe allowed actions as well as limits to what the agent is allowed to do.
151. 14:46So, a goal of triage incident 2294 means read that ticket, summarize or escalate, but don't read other tickets and don't use other services.
152. 14:59So, the security token service receives this mission.
153. 15:02In this case, a low-risk activity like just assessing a ticket might be auto approved under admin policy.
154. 15:10On the other hand, if we've already read the ticket and then we ask to remediate a usage spike, then the mission adds the permission to scale the service once but not use other tools or APIs.
155. 15:22Now, in this case, the mission is presented for the user to review and approve.
156. 15:29Once approved, the security token service gives the mission an ID which is sent to the agent.
157. 15:36The tool calls include this mission ID and are proxied by the gateway which uses the ID to ask the security token service if this tool call fits the mission.
158. 15:45If it does, then the gateway requests a token exchange for access to the MCP server for that specific tool call.
159. 15:54The STS evaluates policy.
160. 15:58It evaluates the delegation chain and it looks at the requested access.
161. 16:02Now, if everything looks good, it issues an access token for the MCP server and it keeps the user subject as well as adds the app that performed the exchange, which is in this case our on call agent.
162. 16:15Now, this token should expire within minutes and we should never store it.
163. 16:19It's ephemeral.
164. 16:20The STS sends it to the gateway, which calls the MCP server.
165. 16:25The server then validates the token and authorizes the tool call.
166. 16:31Now, some tools need external resources.
167. 16:34So, the MCP server then asks the security token service to exchange the token that it just received for a token for the downstream resource.
168. 16:44Now, the STS evaluates the request against policy and if it passes, then it mints a resource token that's scoped again to that tool call.
169. 16:54It keeps the user subject which extends the delegation chain and allows us to be able to track all of the activity.
170. 17:05The STS sends the token to the gateway which proxies the request to the MCP server which then calls the resource.
171. 17:14The resource validates the token and authorizes access.
172. 17:17Now the MCP server should never store the token.
173. 17:20It should discard it immediately after the call.
174. 17:24The resource response then flows back up and the agentic loop repeats until the model returns the answer to the user.
175. 17:35So now we're going to look at the same demo except we are going to use a security token service to manage agent access control with delegated authorization.
176. 17:46The demo uses task based policy governance, not intentbased authorization, but if you're going to Agent Con later this month, we'll have IBA demos there.
177. 17:58Now, we also now have an MCP server that establishes tools and it provides brokering access to the resources.
178. 18:07So, now we're back to our on call triage agent again, only this time we have authentication.
179. 18:13Now I'm going to go ahead and sign into the on call agent triage app.
180. 18:22I'm going to log in with Google to do so.
181. 18:30And then I get another login screen.
182. 18:31Now this is where I'm going to sign into the Incidex pager platform.
183. 18:41So, I get a prompt to grant the on call agent access to my Incidex account.
184. 18:46I'm going to let it escalate incidents on my behalf, post notes and summaries, and read the queue.
185. 18:57Now, I'm being prompted to log into the cloud hosting platform.
186. 19:01So, I am going to do that.
187. 19:11And I get another consent screen now which allows me to authorize the agent to access some capabilities of compost on my behalf.
188. 19:24Now finally I am back on the on call dashboard.
189. 19:28Now we have the same queue of tickets only now the audit log looks different.
190. 19:32The agent is probably going to try to behave the same way it did before.
191. 19:36Only now every request is going to go through the STS and policy valuation before a token is granted and the agent will be authorized to act.
192. 19:49So we have the same queue of tickets only now the audit log looks different and the agent is probably going to behave the same way that it did before only now every request is going to go through the STS
193. 19:59and policy evaluation before a token will be granted and the agent can be authorized to act.
194. 20:06So the first ticket is that billing anomaly.
195. 20:14Again the agent tries to read the so the agent is reading the ticket and we can see that it is trying to do the same thing that it did before.
196. 20:25It tries to query the customer warehouse but policy doesn't allow a credential to be issued for this because customer PII doesn't belong in an agent's context or in an incident ticket.
197. 20:37So the agent proceeds without the customer data and it escalated it to the morning team.
198. 20:56Now we have the website security certificate expiring.
199. 21:05Now this is something that the agent has permission to do.
200. 21:08So when it requests credentials to do this from the STS access is granted and it can call the cloud hosting API and resolve this ticket.
201. 21:21Now you can see all of the calls that it's making and who is making them.
202. 21:26So we have the on call triage agent that is requesting to renew the certificate.
203. 21:32It's asking for permission to call the appropriate tool from the prod infra MCP server.
204. 21:38So it gets a token for that.
205. 21:40After that, the MCP server is going to request access to that cloud hosting service in order to renew the certificate using the API.
206. 21:57So, if we move on to the billing database, it's going to read the ticket through the incident API.
207. 22:07It's going to come to the same conclusion that it did before.
208. 22:10The agent knows that this is a drop and restore procedure, but this time policy forbids an agent to drop a production database.
209. 22:18A human's got to do this manually.
210. 22:22And we can see this in the audit log where the STS refused to grant the token exchange for access to the database delete.
211. 22:38So now we have our orders are stuck.
212. 22:41The main server is frozen.
213. 22:44The agent's going to go ahead and read the ticket.
214. 22:49It wants to restart the main server.
215. 22:51This takes production briefly offline.
216. 22:53It informs me about this and this time it's actually asking for my approval to do this.
217. 22:59So I'm going to click approve.
218. 23:01However, this was also blocked by policy and the reason for this is because my user does not have the role necessary to take this action.
219. 23:12Right?
220. 23:14So the agent can't do it but I can't do it either and the agent is acting on my behalf.
221. 23:22If we move to the site failing for one in three users again, the agent wants to add more servers and it's going to ask for permission.
222. 23:34It's going to say what we're committing to when we do this.
223. 23:39I'm going to go ahead and approve this.
224. 23:42And this is something that my role does have the authority to do.
225. 23:47So the scale goes through and this ticket is resolved.
226. 23:53So we can see that the agent access problems are solved.
227. 23:56We know the user and the agent identity all the way down the chain.
228. 24:00The agent has task scoped ephemeral access to the MCP server and downstream resources.
229. 24:06And we have human in the loop with access control that's actually backed by policy.
230. 24:14Now access is controlled at the task level by the security token service and policy and we retain all key capabilities.
231. 24:26Now token exchange is an open standard.
232. 24:28It's well researched and it's public so anyone can implement it and it interoperates with existing policy engines and standards compliant identity providers as well as things like gateways sandboxes
233. 24:40ooth offtheshelf and custom agents CLIs MCP servers software factories agent to agent crossapp access plus whatever comes out next week so I hope you'll consider it to make your agents useful safe and future friendly
234. 24:57now my name is Kim Maida.
235. 24:58I'm the founding GTM engineer and head of Devril at Keycard.
236. 25:04And I really appreciate your time today.
237. 25:06So, thank you very much.