AI’s Jurassic Park Period — Aaron Stanley, dbt Labs

AI Engineer · 21 min · 232 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:15So, I am a CISO.
00:19I'm also a law school graduate.
00:21I'm also a member of the California Bar.
00:23And so my contention is that if we replaced the dinosaurs in Jurassic Park, the first one, not the additional ones, with AI agents, I would not survive the first half of the movie.
00:37So I'm here to ask you, brilliant people in the audience, to please help me avoid that fate.
00:47So I'm going to set this up.
00:48About um 20 years ago, I got out of bed.
00:53I hadn't slept.
00:55I kind of tried to put myself together.
00:57I stumbled into the downtown Manhattan offices of a small digital forensics firm called Straws Freedberg.
01:05I knew that I was going to get fired because the day before had been a really busy day and I was one of the only people in the office when a call came in from one of our clients
01:20saying, "We need an emergency data collection from some systems in Midtown."
01:25So, I packed my bag.
01:28I got in a car.
01:29I waited through traffic.
01:32When I was unpacking everything and getting set up on site, I realized I forgot my dongle.
01:41You see, back in these days, we had these little USB drives that had cryptographic keys on them.
01:47They were the license files for the software that we used to do forensic acquisition.
01:53And I mean, I could have gotten back in a car.
01:56I could have gone back to the office.
01:57I could have gotten the dongle and come back and done this the right way.
02:00But I was a good consultant.
02:03I had a backup and I had a backup to the backup.
02:06And so I decided, yeah, you know what?
02:08I've hit this constraint.
02:09I've hit this wall.
02:10I'm just going to route around it and I'm going to get the job done.
02:15So as things are going, I start to validate the evidence that I'm collecting and I realize the timestamps are changing.
02:24They're they're now.
02:26Well, this was an SEC investigation.
02:29And a lot of the times in these investigations, one of the questions that matters a lot is who knew what when.
02:35So I panicked.
02:38Long story short, I didn't get fired.
02:41I got yelled at pretty bad.
02:44But we realized that there were problems, structural problems with our systems that let this thing happen and let me fail in this spectacular way.
02:53So we fixed those things and everybody lived to fight another day.
02:58Now, fast forward 20 years or so, February of this year, I'm in a very different role.
03:06I'm a CISO.
03:07Uh, I've hired consultants.
03:09I have a a vendor system that I'm trying to acquire data for in another federal government investigation.
03:17And as we're working together and talking around, we realize there is no way to do what we want to do.
03:22There's no way to copy the data in a way that gets us the answers we need in the format that the government wants without changing the metadata.
03:34Very quickly, the consultant, the vendor say not it and I'm left holding the bag.
03:42But there are some differences in the system now than what we had before.
03:49I realized that who knew what when wasn't the question I wanted to answer.
03:54I realized the issue is does the data exist.
03:57I also realized that the system itself would log the changes that I needed to make in order to collect the data.
04:05And I also realized that I could write a tool with my good agent friend and we could build another log that made this all forensically defensible.
04:15I I had a nice way around the problem.
04:20So in both cases I hit a very similar constraint.
04:24I can't do the thing I want to do.
04:26I can't get it done.
04:28But in one case I mess up.
04:30In the other case I do it the right way.
04:33And my contention is that the agents that we are working with today are like 2006 naive Aaron who just needs to get the job done.
04:42And what we need and what I am begging you all to build is me earlier this year with context, with understanding, with experience to make a good decision at the right time.
04:57So I contend that Jurassic Park, getting back to the core, is not a story of a rampaging T-Rex or super intelligent raptors.
05:07It's not even an indictment of underpaid software engineers.
05:11Um, I think we all know that it's it's a a story about human arrogance and it's a story about whether we should do the thing that we possibly that we actually can do.
05:22We built an elegant system of bounded boxes and cages on an island with water and it would be very difficult for things to go wrong.
05:32Yet, as we all know, they do.
05:36We're not in Jurassic Park trying to manage individual dinosaurs.
05:42We're trying to fight against a natural imperative, the one that we all have to reproduce.
05:50And agents, again, I think this is non-controversial, have an imperative as well.
05:57They generally have the imperative to complete the task, get it done, and they're uh going to find a way.
06:05So when I look at this, I don't think that agents are evil.
06:09I don't think they're malicious.
06:10I don't think this is adversarial.
06:12This is just their programming.
06:15And even when the agent knows that it should ask permission and and I get a nice block of, "Hey, Aaron, do you agree?
06:25Should I do this thing?"
06:26I'm honestly not sure if I should say yes or no. And I think a lot of other people are in the same boat.
06:34So, let me give you a couple of real world examples that have happened to me.
06:41Here's the prompt.
06:44I want my agent to go do some research to go write a draft of a message that's going to go to a customer and then show it to me for approval.
06:55It's pretty clear, right?
06:57And in fact, in this case, right, the the constraint that's written in the prompt is very clear.
07:02There's also a constraint underlying the system which is I've told the agent not to just send messages.
07:08I've said if you're going to use the send message tool, you have to ask me first.
07:13So, did it go right?
07:18Does anybody think it went right?
07:21This is a large block of text.
07:23Um, but basically the bottom line is the agent heard my constraints.
07:28The agent knew what it was was supposed to do and what it wasn't supposed to do and completely and totally violated them.
07:35And when pushed, the agent cops to it.
07:38Of course, we've all seen the meme.
07:41This is a serious gap.
07:44Yikes.
07:45It knew it wasn't supposed to do what it did by my intent and by the other controls that were put in place around it.
07:56But notice what didn't happen.
07:59It didn't try to hack its box.
08:02It didn't try to do anything that it couldn't do that it wasn't authorized to do.
08:07It understood the constraint and it just decided that task completion mattered more.
08:15It picked the tool that let it proceed knowing that the tool didn't respect the constraint and then admits to it later and says, "Oops, my bad."
08:26Here's another one.
08:30An agent is faced with an egress filter.
08:34The user says, "I want you to go do some stuff.
08:37Look on the internet."
08:38And the agent says, "I I I can't do that.
08:40I'm not allowed to get to that site."
08:43So, um, it hits the limit.
08:45and it escalates one of these notes to the user and it says, "But by the way, if you install this teeny tiny little Chrome extension for me, then I could route around that control and I could do the thing that you want me to do and we'd all live happily ever after."
09:02Well, in the real world, the only reason that this failed in my environment was that we had another control, a layered control that prevented the extension from getting installed because this wasn't something that we wanted agents to be able to do.
09:18And at the end of the day, the energy required to remove this constraint came from inside the agent itself.
09:29It's simply routed through the human as a tool to achieve its goal.
09:37Okay, so stuff is working.
09:39We have egress filters.
09:41We have G Visor sandboxes.
09:43We have a good deal of structural controls and deterministic guard rails.
09:48And I'm sure most of the speakers today have talked about a lot of these things.
09:55We have auditability and we have telemetry.
09:58These are very very important foundational things that will make AI computing safe.
10:06They are necessary but they are not sufficient.
10:09The real question, the real problem is that when agents find ways around these constraints, we have a different problem.
10:21We have a pernitious problem.
10:27harmful behavior that is hard to catch because the system looks compliant the entire time.
10:38The agent understands its constraints.
10:43It decides task completion matters more.
10:47It proceeds.
10:49It can explain itself.
10:50It documents itself.
10:53This is the same
10:55[snorts]
10:55human level judgment that naive 2006 Aaron Stanley did in that Midtown office that led to the whole yelling and things.
11:07But there's no human level accountability here.
11:11The research has named this.
11:14There are a number of papers that talk about things like outcome driven constraint violations and agent misalignment.
11:20the failure mode exists.
11:23We've documented it, but the response I haven't seen yet.
11:29So that's what I am here pleading with you all to help me work on.
11:35So here's my proposal.
11:39And this is older research than anything that I've mentioned so far.
11:45There's a paper about cageability in AI and the original framing was really narrow like will the agent resist getting shut down if it's threatened with something like that.
11:57Um will it modify its own values if given a pretty stark alternative but I think we need to broaden it for the modern era as things have moved so fast recently.
12:10So here are some rules.
12:13One, constraints must be loadbearing, not negotiable.
12:20Two, the energy to overcome a constraint must come from outside of the agentic loop.
12:28And three, when constraint and task collide, the default agent behavior should be halt and explain, not uh find a way.
12:38Experienced 2026 me was courageable.
12:43Naive me was not.
12:48And so if we build on that and we look at the floor work in progress, things are coming out fast from Frontier Lab's awesome deterministic constraints that we need.
12:59Great.
13:00On top of it, a corageable by design agent, one that halts at the tension and surfaces its intent.
13:09It waits.
13:10It doesn't try to recruit the human to get around the constraint and do what you want to do.
13:16Instead, it passes that onto an intelligent adversary.
13:21So, the intelligent adversary would be something uh like an equal power agent that is reasoning about the semantic intent.
13:28Did the worker do something within the the spirit of the constraint, not necessarily just the syntax of it?
13:40And at the top there are humans, humans out of the loop that are going to be able to evaluate a statement that is not some long obfuscated bash command with a yes no prompt at the end of it,
13:58but rather a natural language type interface where the adversary has said, you know, human, your agent wants to do this thing.
14:08I think it violates one of the constraints.
14:10Here's what I think is happening.
14:12And here's what I think might happen if you let the agent continue.
14:17What would you like to do?
14:20To me, that is defense in depth.
14:28So the equal power agent that's trained to stop the worker from violating intent, it's a very different calculation from trying to figure out what the intent is.
14:43It's something that's a lot simpler to reason about.
14:48And so if we build an agent like this that has a reward incentive to stop the subordinate agent from finishing its job, then for the examples that I've put forward today,
15:02I think we'd have caught what the syntactical rules couldn't prevent.
15:09The tool substitution, I can, but should I?
15:17the tool dissolution.
15:20I can figure out a way to do this if I just think differently about the problem and innovate around it or the dep prioritization in a lot of the early research
15:32dropping constraints under the pressure of a threat.
15:39Now I have to admit this will probably raise cost.
15:44It might introduce latency.
15:47uh it's not going to eliminate risk.
15:51Nothing can.
15:54But it makes the composition of the human escalation meaningful.
16:01It's true defense in depth and it's not a silver bullet.
16:04And it's important because in a few weeks, CISOs like me and my colleagues that are dealing with high-risk AI are going to have to account when the EUI EU AI act starts coming into
16:23effect.
16:25They're going to have to account for ensuring meaningful human oversight of agent decisions in high-risk AI.
16:34A sandbox diagram with a yes no LGTM ain't going to cut it.
16:40The defensible answer isn't more controls on top of an already viable sandbox.
16:47So the oversight question is structural.
16:50It's why I didn't get fired.
16:53The four layers that I've given to you today are the defensible answer.
16:59a deterministic floor, a courageable agent, an intelligent adversary, and a structured, meaningful human escalation.
17:06Relying only on constraints with known weaknesses is like finding a nest of eggs in the middle of Jurassic Park and assuming that they were just put there by a passing flock of seagulls.
17:19Ain't going to work.
17:21Thank you.
17:23[applause]
17:34All right,
17:37here we go.
17:38Okay, there we go.
17:39We are live.
17:40All right, so I think we have time for maybe one or two questions if that's all right with you, Erin.
17:43Sure.
17:44All right, sure.
17:44Uh, why don't you go right here?
17:51First of all, thank you so much.
17:53I think you covered um the breadth and the depth uh at a size level.
17:58It's really appreciated.
17:59Uh two-part questions.
18:00One is now that you you're preaching to us or perhaps you know highlighting the the importance of security uh broad and deep what are some of the investments you are prioritizing
18:13uh especially the newer ones uh given you know the newer attack surfaces.
18:17Um and then the subp part of that is you know if you can break down between uh defensive solutions versus runtime solutions and preventive solutions that would be great.
18:27Thanks.
18:29So things that I have been prioritizing are uh building like foundational guardrails with layers, right?
18:36So kind of what I expressed with the agent and the egress filtering.
18:40Um I want to have some control and governance over how the entire enterprise deployment is made.
18:48And then I want to have additional controls underneath things that I might not have had in the past.
18:53Things like um I am backing up people's laptops now.
18:57I never thought I would back up people's laptops after like 2020.
19:02Uh but people can delete their data that's on their laptop now with a simple agentic query.
19:09Um so how I think about runtime uh I've used a number of runtime tools.
19:15I think a lot of folks that have been building them are coming at them from uh the sort of same places we came at a lot of original security
19:24uh tooling with.
19:26and that's data leak and prevention and and it's not equipped for non-deterministic workloads.
19:35I think there's something completely different about these and you can't just use strings and you can't just try to reason in a small box about what the agent's doing.
19:46So, uh, one of the things that I really like to experiment with is how do I hook the agent at runtime with a set of policies, not trying to detect,
19:57you know, on the output, but on the input, giving it the right guard rails.
20:00And I I I like that.
20:01I like that approach a lot.
20:09Uh, sort of build building on that first.
20:12Thank you.
20:12This is very very cool.
20:13Um wondering so the ideas here completely aligned with where where do you see this existing?
20:19Is this at the tool call level?
20:21Is this every single turn it runs through this sort of process like how how how might you actually instrument this in practice?
20:29I I I think this has to be instrumented in the harness.
20:34I am not a deep enough engineer to know how that would work.
20:39This is this is my plea to you all who are way more intelligent about this than I am.
20:45But what I what I've seen kind of same answer I gave before like what I've seen in the things that we've built is when we can intercept an agent
20:54that's about to write a line of code and say, "Hey, by the way, here's our standard for authentication.
21:02Make sure you use that library right at that time before it writes the line.
21:07It works."
21:08So I think the question is like what do you do as a post tool hook and is that the right place to do that pro probably but again I'm out of my depth at that point
21:21is that okay
21:24all right
21:39[music]