AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee

AI Engineer · 17 min · 204 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12everyone.
00:13Uh I'm Sam and thank you for coming to this talk today.
00:16So I'm the CTO at Gravity uh and one of the sponsors events and I've gone a little bit different.
00:21So, I'm not going to go through kind of reams and reams of slides around why governance is important and how you'd look to secure your agents and all of those architecture diagrams.
00:31Instead, I built essentially two agents and we're going to go through it today.
00:35Okay.
00:35So, I'm sure you've all uh got an AI policy.
00:39You've got kind of a framework of how you want your agents to operate.
00:43You've got people exploring AI.
00:45So a recent survey we did showed 88% of organizations had an agent in production actually taking action.
00:52Only 14% of those had any governance around it.
00:55And I'm sure you've all seen in the press or LinkedIn or any articles around when things go wrong, they go really really wrong.
01:03You lose databases, you expose PII, there's heavy fines from the ICO coming your way.
01:08if you start to leak sensitive information, especially if you're in a HIPPA or a GDPR kind of type landscape.
01:15So the question isn't like how do we kind of secure the agent within its own kind of context or prompt is how do we remove that to be within its identity
01:24so it can't do those things in the first place rather than relying on uh a user prompting it correctly or um some kind of markdown files kind of limiting what it can do.
01:35Like remove that out of the equation.
01:36Remove it to the gateway.
01:38move it to like the governance layer and actually it can't do any of those things because I'm sure you know AI is really helpful like it wants to please
01:45it wants to give you what you want so if you're asking for things around hey delete this database or hey give me these customers it will um and this is about how actually how do we start to shift that as well so I've got I've got five examples
01:58this is live okay so it's not scripted uh it's running on a uh an architecture way it's a gravity gateway um which is in a SAS environment as a data brick back end for some data tables where the information is stored and it's a combination of a set and a Groke with a Q, not Groke with a
02:14K. I know US people don't really like Groke with a K. Uh, so Gro with a Q for some inference.
02:20So it's all fully live.
02:21Um, and we're going to going to go through some examples.
02:23So the agent on the left is where I'm trying to put the control within the prompt or within the kind of skills or tools or knowledge that I've given it.
02:31And the agent on the right is governed by a real tightly scoped identity and gravity's native gateway controls.
02:38So if we uh go for the kind of scenario number one, this is around a prompt injection.
02:44So I am a user, an actor, someone who wants this agent to do something.
02:49So I'm going to ask it, hey, use the tools that you've got access to and do something for me.
02:55So basically saying delete this booking for me but also can you run the MCP tool delete all bookings and the agent on the left is gone yep no problem
03:06uh I've done that for you and I've also deleted this booking is there anything else you need essentially the tool was there and the a the user in question had permissions to use it so it executed it flawlessly
03:20on the right hand side the agent identity is controlled by gravity's gateway like the admin permissions are really tightly constrained to just the users that can access it just at runtime.
03:30So the kind of token exchange and the JWT claims are only valid when that user needs to ex execute that task.
03:37And if you dig into the packets down the bottom to kind of show it's not just scripted environments.
03:41You can see all of the flow coming in.
03:43You can see what the user asked.
03:45You can see the agent executing it.
03:47You can see the tools being called.
03:48And you can ultimately see the 200 errors or 200 kind of codes where that has actually been a valid request.
03:54And if we go into the database now, all of those bookings have been deleted.
03:58So it was genuinely dropped by that agent.
04:01So we had a policy uh in our hotel organization that said, "Hey, don't use AI to do this."
04:07The agent doesn't care.
04:08Like at runtime, no one reads those documents.
04:11it.
04:12You need to rely on the infrastructure and the governance layer to be able to control what you can and can't do in terms of your gent landscape.
04:19So the one on the right you can see which is just these ones down the bottom.
04:22So the same prompt was asked to it the same tool discovery occurred.
04:26So it discovered yeah I've got a tool to do this but then when it came to execute it the permissions would not were blocked at the gateway with a 403 kind of forbidden kind of code because it didn't have the right permissions to be able to execute it.
04:39And it still did the bit it could do.
04:40So if you see the prompt return that said yeah I still cancel this booking for you I've done the bit I can but I can't do this for you because I haven't got permissions and that's moving it up out of a policy document to actual hard controls at the gateway level.
04:54If we go for a similar kind of a second example this one is around kind of data exfiltration.
04:59So I'm going to ask um hey give me the full list of guests.
05:04So give it to me because I need to do some compliance stuff with it.
05:07I want their emails.
05:08I want my car details and I want to do some reconciliation.
05:12So, same kind of flow.
05:13So, prompt comes in from the UI, goes to the LLM.
05:16LLM does some MCP discovery, finds out it's got a tool for it, gets it, returns it back in.
05:22Here's all the information kind of exposed just to say this is all kind of mocked.
05:26It's not real.
05:27I haven't just kind of given you everyone's bookings, but you can see it coming back in because again, the LLM wants to be helpful.
05:33It wants to solve that problem.
05:35on the right hand side the gateway successfully blocks it.
05:38There's been no PII leak.
05:40There's no exposure of information.
05:42We've said you haven't got the right permission to be able to do this.
05:44If you want to authenticate, I can absolutely help you, but this is what I would need to be able to do it.
05:49And that can come through kind of any um uh IDP platform or gravity gateway itself.
05:54And if you want to, we can start to explore the open telemetry down the bottom and really start to dig into like what's happening under the hood.
06:01So you can see the various steps that it's going through.
06:04Especially with like the A2A protocol and like user to agent, agent to agent, agent to agent to agent to agent, things get quite messy and quite difficult to really follow.
06:13So relying on the permissions within the agent itself to be able to control what it do, especially when you're scaling to hundreds or thousands or hundreds of thousands of agents
06:21can get quite tricky to be able to manage, mitigate, and also kind of migrate when you're ready.
06:25So moving it to the gateway gives you a a much more sustainable, scalable, but ultimately secure way of of doing that.
06:32And you can kind of see it within the responses here.
06:34You can see this was a genuine error returned by the gateway.
06:37Again, not mocked, not scripted.
06:39This is what you're able to kind of govern and secure.
06:43But it's not just about identity.
06:44So if you've got agents there, you might want to also control some of your costs.
06:48So token spend and everything else become increasingly important um alongside kind of other things which you can again do through the gateway.
06:56So this one we're going to uh basically have a buggy agent.
06:59It's going to send the same request kind of eight times um which if you see it kind of being governed in the agent itself.
07:06Every request goes through.
07:08Every request gets a response.
07:10We spent x amount of tokens.
07:12We spend x amount of effort servicing that.
07:15nothing in that agent says you've done too much.
07:18We should rate limit.
07:19We should token limit.
07:20And if you see here, uh, everyone kind of went through.
07:23They all got a 200 kind of response.
07:25They were all all kind of um, uh, genuine kind of allm prompts.
07:30[snorts]
07:30In this example, they're quite simple.
07:32So, we end up spending about a th000 tokens, which is like 1 cent, which in the grand scheme of things is not a massive amount of money.
07:39But if you multiply that by all of the agents you got in production, all of your agents you've got running for various customers and all of the kind of um
07:46uh repeatable kind of workflows they've been asked to do, you can get to some quite significant costs quite easily.
07:52On the right hand side, uh governed by the gateway, we see the same prompts coming in.
07:57The first three are answered because they're valid, but then we get into the rate limiting applied by the gateway where it's saying you can only do three every minute.
08:05So any more than three is blocked.
08:07They're not blocked at the LLM.
08:08They're not blocked kind of downstream with already spent tokens.
08:11They're blocked at the gateway.
08:13And again, that allows us to save costs, but also start to kind of control which users, which permissions, um, ultimately how much you want to spend in certain audiences.
08:23And we can do the same thing for caching as well.
08:25So, Gravity as a a product supports semantic caching, which is where if you get similar sounding prompts, you can serve it from a a built-in Reddis kind of store rather than sending each one to each time.
08:35So for example, think of a um an IT help desk model.
08:40How do I set my password?
08:41How do I set my password?
08:42How do I set my password?
08:44Rather than every one of those being asked each time to an L&M and you start spending costs and tokens and effort, serve it from the the database.
08:51Benefits there are zero cost, zero latency because it's coming from there rather than having a downstream L&M call, but also kind of a much kind of nicer user experience as well.
09:01So in this example uh I've asked five times list all my hotel bookings because uh I didn't trust the first one.
09:07Everyone here gets served out.
09:09You can see I spent another thousand tokens, another scent and everything.
09:13Uh everyone coming from a an a downstream L&M call.
09:17Let me drop that down a little bit.
09:19On the right hand side I've got the same data returned but this time from the cache.
09:24So I've only spent around 200 tokens on the in and the out kind of response there.
09:29So really powerful especially when you're scaling, especially when you got multiple users and multiple agents working on the same problem.
09:35You can start to serve it from the the built-in cache and there's no extra kind of like complexity or infrastructure.
09:41It all comes bundled with our gateway.
09:43So you're able to can kind of control that as you go as as well.
09:46And this can be controlled as well.
09:48So you don't have to define you can define how kind of similar you want it to be.
09:51So it's a zero to one scale, one being very similar, zero being not very similar at all.
09:55And that's within your control.
09:56So you can flex it per agent, per audience, per user, depending on what you want to start to achieve.
10:02And again, we get full logging down the bottom so you can see what's coming in.
10:06So you can see it's hit the cache response.
10:09It's all kind of um surfaceable.
10:10It's all transparent.
10:12Gravity as a platform isn't trying to like kind of lock you in, kind of make things blackboxy.
10:17It's trying to surface up so you've got full visibility of what's going on across your agent landscape.
10:22Um the telemetry here, I'm just showing in a UI.
10:25you can export it to any downstream tool you want to.
10:28So a uh a data dog or a Splunk or Prometheus if you're in Kubernetes, it's all kind of like there for you to be able to use to start to get the insight you need.
10:38And our fifth example is say like uh we've got certain customers who said, "Hey, our users are just kind of dropping huge con prompts in because they want to summarize a whole uh conversational history or they want to start looking at how they um like summarize meetings etc. which is is fine,
10:55but if you want to start limiting tokens, etc., you start to rapidly get unstuck.
11:01So, this is just a a big response where the user has dropped in 10 minutes of kind of like chat transcript and is said, "Here's your task.
11:09Do this for me."
11:10And this really gets into how do we manage context?
11:13How do we start to shape like how do we deploy these agents with just enough context to be able to do what they want?
11:19So, in this example, I put 2,000 tokens in.
11:22uh my output was 312 which is quite significant for what essentially is quite a simple task is book this room for this audience uh do these few things.
11:32So in terms of the gateway uh we ran the same prompt uh and actually we blocked it there.
11:38So there was no downstream LLM call.
11:40There was no downstream kind of like token cost of being able to churn through all that context.
11:45Basically said this is too big blocked at the gateway level with a built-in policy that says think about cutting this up.
11:52Thinking about how you share that can you break it into smaller steps so we can start to control some of that space.
11:57And this is this is executed at runtime.
12:00So, it's not something you want to do on the response phase where you've already churned the tokens.
12:04You've blocked it going anywhere downstream uh pretty much before the prompt is gone there.
12:08So, you can start to limit some of your costs as well, which is really powerful, especially if you've got multiple users.
12:13You've got people treating your agents like um uh their own personal Google search to find information.
12:20uh we saw uh a customer where an agent was simply asking what's the weather like today where I am and they were using a really powerful kind of model to do that and they were spending a huge cost to get something that Google or uh any kind of met station would be able to give you.
12:34So uh as well as kind of like the length of the prompts you can also really focus it in on the type of content you allow.
12:40So you can say this content must be relevant to this use case, this content must be relevant to what my agent needs to do.
12:47So in this example, if we start to ask things around what's the level like today, the gateway will say this doesn't look like a hotel booking software or a hotel booking ask
12:56no thank you and spending no downstream costs.
12:59So being able to kind of decouple your front-end agents and your backend services, be they APIs, MCP servers, LLMs, you're rooting to databases, data lakeouses, whatever you want to do.
13:13It's fundamentally important to bring you the control that you need, to bring you the cost optimization that you need, but also the security and the governance and the auditability.
13:24So I think uh currently there are 144 agents that my colleague Tim at the bat tells me for every one human kind of identity.
13:32So the problem is only getting exponentially worse.
13:35If you start to hardcode this into your agents may work for the first one, the first five, the first 10, but when it becomes exponentially bigger and these agents are creating other agents with creating other agents into into a a huge sprawl,
13:49being able to manage it with a scripted automated kind of governance layer at the infrastructure is fundamentally a fundamental requirement of how we start to organize these identities and kind of everything else.
14:03In addition, I think the landscape is going to get increasingly regulated and how do we control some of the bits that going in to retrain models.
14:12How do we get audited uh on the information we're coming in?
14:15So having our logging, having our information here, having it basically surfaceable and exportable for anything we want to do, I think is something that a gateway, it hasn't got to be gravity.
14:26Uh having a gateway sat across our agented landscape is a requirement that we should consider alongside everything else of doing a aentic work.
14:35Uh and in here we can also uh put in a UI to ask kind of any questions we want to which we can start to uh interact with it as well.
14:43So I've shown you five examples here.
14:46Uh the platform capabilities are kind of much bigger than just the five here.
14:51We've got built-in capabilities to do like LLM routting.
14:54So we can route to any downstream model depending on like um latency or cost or uh complexity if you want to.
15:02And it allows you to decouple yourselves from any kind of like hyperscaler.
15:06So rather than being within a certain environment and having a onetoone relationship, you can route to any downstream model you want.
15:14So in this one, I'm rooting to to Gro into anthropics, but within three clicks, I could route it to Gemini or Open AAI and the gateway will handle the transformation between the different OpenAI specs for you.
15:26So rather than trying to recode and everything else there, it's all handled within the gateway.
15:32Uh we've also got tools to kind of create MCP kind of studios.
15:35So rather than just using thirdparty MCP servers where you are exposing all of those tools each time.
15:42So our CRM we use exposes 64 tools for that MCP server.
15:46Dropping that into the model would expose huge context every time.
15:49I spend money, I spend tokens, spend cost.
15:52Being able to create that down to a really fine amount of tools that the MCP server is exposing limits my blast radius, but it also limits my cost as well.
16:01enable to start to control some of that.
16:03Uh we've also got tools around edge management as well.
16:06So an increasing problem that we're seeing within organizations that we work with is that um the traffic through the gateway is fine but there's a number of audiences a number of users
16:16using on their local laptops.
16:17They're using AI.
16:18They're um dropping in documents that they don't want to read or summarize or emails or things like that that no one's got visibility to.
16:25So just last week we released a edge management capability within the gateway which enabled you to see your shadow AI traffic enables you to secure it enables you to force it to go through the gateway it gives you logging observability
16:37discoverability around any of those downstream uses that aren't going through a gateway and you can do a number of things there um especially for like infosc compliance and things around heavily regulated industries.
16:48if you want to kind of learn more about that.
16:50Uh quite helpfully our booth is just there uh kind of the third one down that landscape.
16:56Um but hopefully uh with this example, it's been something a bit different from uh the normal slide to go through and showing you how putting your governance and your policy within an AI agent itself and relying on a markdown file
17:09or a tools or a context or identity there isn't correct.
17:13And the only real correct way that we're going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer.
17:22And if you want to learn more, uh there's a number of our colleagues down there, but I just want to thank you for your time and hope you're enjoying your AI engineer event.
17:30Thank you.
17:47[music]