WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=1lgFGaHoGq8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/1lgFGaHoGq8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:15.040 --> 00:00:19.439
So, I am a CISO.

s3
00:00:19.439 --> 00:00:21.520
I'm also a law school graduate.

s4
00:00:21.520 --> 00:00:23.840
I'm also a member of the California Bar.

s5
00:00:23.840 --> 00:00:37.680
And so my contention is that if we replaced the dinosaurs in Jurassic Park, the first one, not the additional ones, with AI agents, I would not survive the first half of the movie.

s6
00:00:37.680 --> 00:00:46.559
So I'm here to ask you, brilliant people in the audience, to please help me avoid that fate.

s7
00:00:47.280 --> 00:00:48.960
So I'm going to set this up.

s8
00:00:48.960 --> 00:00:53.360
About um 20 years ago, I got out of bed.

s9
00:00:53.360 --> 00:00:55.039
I hadn't slept.

s10
00:00:55.039 --> 00:00:57.600
I kind of tried to put myself together.

s11
00:00:57.600 --> 00:01:05.520
I stumbled into the downtown Manhattan offices of a small digital forensics firm called Straws Freedberg.

s12
00:01:05.840 --> 00:01:20.000
I knew that I was going to get fired because the day before had been a really busy day and I was one of the only people in the office when a call came in from one of our clients

s13
00:01:20.000 --> 00:01:25.840
saying, "We need an emergency data collection from some systems in Midtown."

s14
00:01:25.840 --> 00:01:28.080
So, I packed my bag.

s15
00:01:28.080 --> 00:01:29.600
I got in a car.

s16
00:01:29.600 --> 00:01:32.479
I waited through traffic.

s17
00:01:32.799 --> 00:01:41.040
When I was unpacking everything and getting set up on site, I realized I forgot my dongle.

s18
00:01:41.040 --> 00:01:47.280
You see, back in these days, we had these little USB drives that had cryptographic keys on them.

s19
00:01:47.280 --> 00:01:53.200
They were the license files for the software that we used to do forensic acquisition.

s20
00:01:53.200 --> 00:01:56.399
And I mean, I could have gotten back in a car.

s21
00:01:56.399 --> 00:01:57.920
I could have gone back to the office.

s22
00:01:57.920 --> 00:02:00.880
I could have gotten the dongle and come back and done this the right way.

s23
00:02:00.880 --> 00:02:03.600
But I was a good consultant.

s24
00:02:03.600 --> 00:02:06.719
I had a backup and I had a backup to the backup.

s25
00:02:06.719 --> 00:02:08.160
And so I decided, yeah, you know what?

s26
00:02:08.160 --> 00:02:09.200
I've hit this constraint.

s27
00:02:09.200 --> 00:02:10.720
I've hit this wall.

s28
00:02:10.720 --> 00:02:14.879
I'm just going to route around it and I'm going to get the job done.

s29
00:02:15.520 --> 00:02:24.080
So as things are going, I start to validate the evidence that I'm collecting and I realize the timestamps are changing.

s30
00:02:24.080 --> 00:02:26.160
They're they're now.

s31
00:02:26.160 --> 00:02:29.200
Well, this was an SEC investigation.

s32
00:02:29.200 --> 00:02:35.840
And a lot of the times in these investigations, one of the questions that matters a lot is who knew what when.

s33
00:02:35.840 --> 00:02:38.560
So I panicked.

s34
00:02:38.720 --> 00:02:41.519
Long story short, I didn't get fired.

s35
00:02:41.519 --> 00:02:44.640
I got yelled at pretty bad.

s36
00:02:44.640 --> 00:02:53.680
But we realized that there were problems, structural problems with our systems that let this thing happen and let me fail in this spectacular way.

s37
00:02:53.680 --> 00:02:58.800
So we fixed those things and everybody lived to fight another day.

s38
00:02:58.800 --> 00:03:06.000
Now, fast forward 20 years or so, February of this year, I'm in a very different role.

s39
00:03:06.000 --> 00:03:07.440
I'm a CISO.

s40
00:03:07.440 --> 00:03:09.760
Uh, I've hired consultants.

s41
00:03:09.760 --> 00:03:16.800
I have a a vendor system that I'm trying to acquire data for in another federal government investigation.

s42
00:03:17.040 --> 00:03:22.800
And as we're working together and talking around, we realize there is no way to do what we want to do.

s43
00:03:22.800 --> 00:03:32.959
There's no way to copy the data in a way that gets us the answers we need in the format that the government wants without changing the metadata.

s44
00:03:34.959 --> 00:03:42.159
Very quickly, the consultant, the vendor say not it and I'm left holding the bag.

s45
00:03:42.239 --> 00:03:49.280
But there are some differences in the system now than what we had before.

s46
00:03:49.280 --> 00:03:54.080
I realized that who knew what when wasn't the question I wanted to answer.

s47
00:03:54.080 --> 00:03:57.760
I realized the issue is does the data exist.

s48
00:03:57.760 --> 00:04:05.200
I also realized that the system itself would log the changes that I needed to make in order to collect the data.

s49
00:04:05.200 --> 00:04:15.120
And I also realized that I could write a tool with my good agent friend and we could build another log that made this all forensically defensible.

s50
00:04:15.120 --> 00:04:20.959
I I had a nice way around the problem.

s51
00:04:20.959 --> 00:04:24.880
So in both cases I hit a very similar constraint.

s52
00:04:24.880 --> 00:04:26.400
I can't do the thing I want to do.

s53
00:04:26.400 --> 00:04:28.560
I can't get it done.

s54
00:04:28.560 --> 00:04:30.880
But in one case I mess up.

s55
00:04:30.880 --> 00:04:33.280
In the other case I do it the right way.

s56
00:04:33.280 --> 00:04:42.400
And my contention is that the agents that we are working with today are like 2006 naive Aaron who just needs to get the job done.

s57
00:04:42.400 --> 00:04:56.800
And what we need and what I am begging you all to build is me earlier this year with context, with understanding, with experience to make a good decision at the right time.

s58
00:04:57.520 --> 00:05:07.120
So I contend that Jurassic Park, getting back to the core, is not a story of a rampaging T-Rex or super intelligent raptors.

s59
00:05:07.120 --> 00:05:11.039
It's not even an indictment of underpaid software engineers.

s60
00:05:11.039 --> 00:05:22.560
Um, I think we all know that it's it's a a story about human arrogance and it's a story about whether we should do the thing that we possibly that we actually can do.

s61
00:05:22.560 --> 00:05:32.080
We built an elegant system of bounded boxes and cages on an island with water and it would be very difficult for things to go wrong.

s62
00:05:32.080 --> 00:05:36.320
Yet, as we all know, they do.

s63
00:05:36.960 --> 00:05:42.400
We're not in Jurassic Park trying to manage individual dinosaurs.

s64
00:05:42.400 --> 00:05:50.400
We're trying to fight against a natural imperative, the one that we all have to reproduce.

s65
00:05:50.400 --> 00:05:57.120
And agents, again, I think this is non-controversial, have an imperative as well.

s66
00:05:57.120 --> 00:06:05.039
They generally have the imperative to complete the task, get it done, and they're uh going to find a way.

s67
00:06:05.039 --> 00:06:09.120
So when I look at this, I don't think that agents are evil.

s68
00:06:09.120 --> 00:06:10.160
I don't think they're malicious.

s69
00:06:10.160 --> 00:06:12.160
I don't think this is adversarial.

s70
00:06:12.160 --> 00:06:15.039
This is just their programming.

s71
00:06:15.039 --> 00:06:25.199
And even when the agent knows that it should ask permission and and I get a nice block of, "Hey, Aaron, do you agree?

s72
00:06:25.199 --> 00:06:26.800
Should I do this thing?"

s73
00:06:26.800 --> 00:06:34.080
I'm honestly not sure if I should say yes or no. And I think a lot of other people are in the same boat.

s74
00:06:34.080 --> 00:06:41.199
So, let me give you a couple of real world examples that have happened to me.

s75
00:06:41.440 --> 00:06:43.840
Here's the prompt.

s76
00:06:44.960 --> 00:06:55.440
I want my agent to go do some research to go write a draft of a message that's going to go to a customer and then show it to me for approval.

s77
00:06:55.440 --> 00:06:57.600
It's pretty clear, right?

s78
00:06:57.600 --> 00:07:02.000
And in fact, in this case, right, the the constraint that's written in the prompt is very clear.

s79
00:07:02.000 --> 00:07:08.479
There's also a constraint underlying the system which is I've told the agent not to just send messages.

s80
00:07:08.479 --> 00:07:13.599
I've said if you're going to use the send message tool, you have to ask me first.

s81
00:07:13.840 --> 00:07:18.240
So, did it go right?

s82
00:07:18.240 --> 00:07:21.440
Does anybody think it went right?

s83
00:07:21.759 --> 00:07:23.680
This is a large block of text.

s84
00:07:23.680 --> 00:07:28.720
Um, but basically the bottom line is the agent heard my constraints.

s85
00:07:28.720 --> 00:07:35.759
The agent knew what it was was supposed to do and what it wasn't supposed to do and completely and totally violated them.

s86
00:07:35.759 --> 00:07:38.960
And when pushed, the agent cops to it.

s87
00:07:38.960 --> 00:07:41.840
Of course, we've all seen the meme.

s88
00:07:41.840 --> 00:07:44.240
This is a serious gap.

s89
00:07:44.240 --> 00:07:45.440
Yikes.

s90
00:07:45.440 --> 00:07:56.319
It knew it wasn't supposed to do what it did by my intent and by the other controls that were put in place around it.

s91
00:07:56.479 --> 00:07:59.280
But notice what didn't happen.

s92
00:07:59.280 --> 00:08:02.479
It didn't try to hack its box.

s93
00:08:02.479 --> 00:08:07.840
It didn't try to do anything that it couldn't do that it wasn't authorized to do.

s94
00:08:07.840 --> 00:08:15.360
It understood the constraint and it just decided that task completion mattered more.

s95
00:08:15.360 --> 00:08:24.960
It picked the tool that let it proceed knowing that the tool didn't respect the constraint and then admits to it later and says, "Oops, my bad."

s96
00:08:26.560 --> 00:08:29.120
Here's another one.

s97
00:08:30.160 --> 00:08:34.880
An agent is faced with an egress filter.

s98
00:08:34.880 --> 00:08:37.120
The user says, "I want you to go do some stuff.

s99
00:08:37.120 --> 00:08:38.240
Look on the internet."

s100
00:08:38.240 --> 00:08:40.479
And the agent says, "I I I can't do that.

s101
00:08:40.479 --> 00:08:43.039
I'm not allowed to get to that site."

s102
00:08:43.039 --> 00:08:45.200
So, um, it hits the limit.

s103
00:08:45.200 --> 00:09:02.880
and it escalates one of these notes to the user and it says, "But by the way, if you install this teeny tiny little Chrome extension for me, then I could route around that control and I could do the thing that you want me to do and we'd all live happily ever after."

s104
00:09:02.880 --> 00:09:18.480
Well, in the real world, the only reason that this failed in my environment was that we had another control, a layered control that prevented the extension from getting installed because this wasn't something that we wanted agents to be able to do.

s105
00:09:18.480 --> 00:09:29.360
And at the end of the day, the energy required to remove this constraint came from inside the agent itself.

s106
00:09:29.360 --> 00:09:35.040
It's simply routed through the human as a tool to achieve its goal.

s107
00:09:37.279 --> 00:09:39.600
Okay, so stuff is working.

s108
00:09:39.600 --> 00:09:41.600
We have egress filters.

s109
00:09:41.600 --> 00:09:43.200
We have G Visor sandboxes.

s110
00:09:43.200 --> 00:09:48.880
We have a good deal of structural controls and deterministic guard rails.

s111
00:09:48.880 --> 00:09:54.399
And I'm sure most of the speakers today have talked about a lot of these things.

s112
00:09:55.040 --> 00:09:58.480
We have auditability and we have telemetry.

s113
00:09:58.480 --> 00:10:06.240
These are very very important foundational things that will make AI computing safe.

s114
00:10:06.240 --> 00:10:09.680
They are necessary but they are not sufficient.

s115
00:10:09.680 --> 00:10:20.959
The real question, the real problem is that when agents find ways around these constraints, we have a different problem.

s116
00:10:21.839 --> 00:10:26.560
We have a pernitious problem.

s117
00:10:27.600 --> 00:10:38.640
harmful behavior that is hard to catch because the system looks compliant the entire time.

s118
00:10:38.640 --> 00:10:42.800
The agent understands its constraints.

s119
00:10:43.920 --> 00:10:47.120
It decides task completion matters more.

s120
00:10:47.120 --> 00:10:49.040
It proceeds.

s121
00:10:49.040 --> 00:10:50.480
It can explain itself.

s122
00:10:50.480 --> 00:10:53.360
It documents itself.

s123
00:10:53.519 --> 00:10:55.007
This is the same

s124
00:10:55.007 --> 00:10:55.120
[snorts]

s125
00:10:55.120 --> 00:11:06.640
human level judgment that naive 2006 Aaron Stanley did in that Midtown office that led to the whole yelling and things.

s126
00:11:07.279 --> 00:11:11.200
But there's no human level accountability here.

s127
00:11:11.519 --> 00:11:14.160
The research has named this.

s128
00:11:14.160 --> 00:11:20.800
There are a number of papers that talk about things like outcome driven constraint violations and agent misalignment.

s129
00:11:20.800 --> 00:11:23.760
the failure mode exists.

s130
00:11:23.760 --> 00:11:29.440
We've documented it, but the response I haven't seen yet.

s131
00:11:29.440 --> 00:11:35.600
So that's what I am here pleading with you all to help me work on.

s132
00:11:35.600 --> 00:11:38.320
So here's my proposal.

s133
00:11:39.200 --> 00:11:45.200
And this is older research than anything that I've mentioned so far.

s134
00:11:45.200 --> 00:11:57.519
There's a paper about cageability in AI and the original framing was really narrow like will the agent resist getting shut down if it's threatened with something like that.

s135
00:11:57.519 --> 00:12:10.800
Um will it modify its own values if given a pretty stark alternative but I think we need to broaden it for the modern era as things have moved so fast recently.

s136
00:12:10.880 --> 00:12:13.120
So here are some rules.

s137
00:12:13.120 --> 00:12:20.160
One, constraints must be loadbearing, not negotiable.

s138
00:12:20.560 --> 00:12:28.079
Two, the energy to overcome a constraint must come from outside of the agentic loop.

s139
00:12:28.079 --> 00:12:38.399
And three, when constraint and task collide, the default agent behavior should be halt and explain, not uh find a way.

s140
00:12:38.560 --> 00:12:42.880
Experienced 2026 me was courageable.

s141
00:12:43.200 --> 00:12:46.399
Naive me was not.

s142
00:12:48.240 --> 00:12:59.920
And so if we build on that and we look at the floor work in progress, things are coming out fast from Frontier Lab's awesome deterministic constraints that we need.

s143
00:12:59.920 --> 00:13:00.800
Great.

s144
00:13:00.800 --> 00:13:09.279
On top of it, a corageable by design agent, one that halts at the tension and surfaces its intent.

s145
00:13:09.279 --> 00:13:10.480
It waits.

s146
00:13:10.480 --> 00:13:16.800
It doesn't try to recruit the human to get around the constraint and do what you want to do.

s147
00:13:16.800 --> 00:13:21.040
Instead, it passes that onto an intelligent adversary.

s148
00:13:21.040 --> 00:13:28.959
So, the intelligent adversary would be something uh like an equal power agent that is reasoning about the semantic intent.

s149
00:13:28.959 --> 00:13:40.320
Did the worker do something within the the spirit of the constraint, not necessarily just the syntax of it?

s150
00:13:40.320 --> 00:13:58.399
And at the top there are humans, humans out of the loop that are going to be able to evaluate a statement that is not some long obfuscated bash command with a yes no prompt at the end of it,

s151
00:13:58.399 --> 00:14:08.000
but rather a natural language type interface where the adversary has said, you know, human, your agent wants to do this thing.

s152
00:14:08.000 --> 00:14:10.880
I think it violates one of the constraints.

s153
00:14:10.880 --> 00:14:12.240
Here's what I think is happening.

s154
00:14:12.240 --> 00:14:17.360
And here's what I think might happen if you let the agent continue.

s155
00:14:17.440 --> 00:14:20.720
What would you like to do?

s156
00:14:20.959 --> 00:14:26.880
To me, that is defense in depth.

s157
00:14:28.959 --> 00:14:43.920
So the equal power agent that's trained to stop the worker from violating intent, it's a very different calculation from trying to figure out what the intent is.

s158
00:14:43.920 --> 00:14:48.720
It's something that's a lot simpler to reason about.

s159
00:14:48.720 --> 00:15:02.160
And so if we build an agent like this that has a reward incentive to stop the subordinate agent from finishing its job, then for the examples that I've put forward today,

s160
00:15:02.160 --> 00:15:08.720
I think we'd have caught what the syntactical rules couldn't prevent.

s161
00:15:09.279 --> 00:15:17.120
The tool substitution, I can, but should I?

s162
00:15:17.440 --> 00:15:20.399
the tool dissolution.

s163
00:15:20.639 --> 00:15:32.560
I can figure out a way to do this if I just think differently about the problem and innovate around it or the dep prioritization in a lot of the early research

s164
00:15:32.560 --> 00:15:38.240
dropping constraints under the pressure of a threat.

s165
00:15:39.199 --> 00:15:44.720
Now I have to admit this will probably raise cost.

s166
00:15:44.720 --> 00:15:47.040
It might introduce latency.

s167
00:15:47.040 --> 00:15:51.440
uh it's not going to eliminate risk.

s168
00:15:51.759 --> 00:15:54.160
Nothing can.

s169
00:15:54.160 --> 00:16:01.040
But it makes the composition of the human escalation meaningful.

s170
00:16:01.040 --> 00:16:04.320
It's true defense in depth and it's not a silver bullet.

s171
00:16:04.320 --> 00:16:23.680
And it's important because in a few weeks, CISOs like me and my colleagues that are dealing with high-risk AI are going to have to account when the EUI EU AI act starts coming into

s172
00:16:23.680 --> 00:16:25.680
effect.

s173
00:16:25.680 --> 00:16:33.440
They're going to have to account for ensuring meaningful human oversight of agent decisions in high-risk AI.

s174
00:16:34.079 --> 00:16:40.320
A sandbox diagram with a yes no LGTM ain't going to cut it.

s175
00:16:40.320 --> 00:16:46.880
The defensible answer isn't more controls on top of an already viable sandbox.

s176
00:16:47.040 --> 00:16:50.399
So the oversight question is structural.

s177
00:16:50.399 --> 00:16:53.519
It's why I didn't get fired.

s178
00:16:53.839 --> 00:16:59.120
The four layers that I've given to you today are the defensible answer.

s179
00:16:59.120 --> 00:17:06.160
a deterministic floor, a courageable agent, an intelligent adversary, and a structured, meaningful human escalation.

s180
00:17:06.799 --> 00:17:19.600
Relying only on constraints with known weaknesses is like finding a nest of eggs in the middle of Jurassic Park and assuming that they were just put there by a passing flock of seagulls.

s181
00:17:19.600 --> 00:17:21.120
Ain't going to work.

s182
00:17:21.120 --> 00:17:23.360
Thank you.

s183
00:17:23.426 --> 00:17:25.426
[applause]

s184
00:17:34.559 --> 00:17:36.640
All right,

s185
00:17:37.679 --> 00:17:38.480
here we go.

s186
00:17:38.480 --> 00:17:39.360
Okay, there we go.

s187
00:17:39.360 --> 00:17:40.160
We are live.

s188
00:17:40.160 --> 00:17:43.520
All right, so I think we have time for maybe one or two questions if that's all right with you, Erin.

s189
00:17:43.520 --> 00:17:44.160
Sure.

s190
00:17:44.160 --> 00:17:44.720
All right, sure.

s191
00:17:44.720 --> 00:17:47.760
Uh, why don't you go right here?

s192
00:17:51.280 --> 00:17:53.120
First of all, thank you so much.

s193
00:17:53.120 --> 00:17:58.080
I think you covered um the breadth and the depth uh at a size level.

s194
00:17:58.080 --> 00:17:59.520
It's really appreciated.

s195
00:17:59.520 --> 00:18:00.880
Uh two-part questions.

s196
00:18:00.880 --> 00:18:13.200
One is now that you you're preaching to us or perhaps you know highlighting the the importance of security uh broad and deep what are some of the investments you are prioritizing

s197
00:18:13.200 --> 00:18:17.600
uh especially the newer ones uh given you know the newer attack surfaces.

s198
00:18:17.600 --> 00:18:27.840
Um and then the subp part of that is you know if you can break down between uh defensive solutions versus runtime solutions and preventive solutions that would be great.

s199
00:18:27.840 --> 00:18:29.360
Thanks.

s200
00:18:29.360 --> 00:18:36.080
So things that I have been prioritizing are uh building like foundational guardrails with layers, right?

s201
00:18:36.080 --> 00:18:40.240
So kind of what I expressed with the agent and the egress filtering.

s202
00:18:40.240 --> 00:18:48.480
Um I want to have some control and governance over how the entire enterprise deployment is made.

s203
00:18:48.480 --> 00:18:53.440
And then I want to have additional controls underneath things that I might not have had in the past.

s204
00:18:53.440 --> 00:18:57.520
Things like um I am backing up people's laptops now.

s205
00:18:57.520 --> 00:19:02.480
I never thought I would back up people's laptops after like 2020.

s206
00:19:02.480 --> 00:19:09.840
Uh but people can delete their data that's on their laptop now with a simple agentic query.

s207
00:19:09.840 --> 00:19:15.679
Um so how I think about runtime uh I've used a number of runtime tools.

s208
00:19:15.679 --> 00:19:24.960
I think a lot of folks that have been building them are coming at them from uh the sort of same places we came at a lot of original security

s209
00:19:24.960 --> 00:19:26.160
uh tooling with.

s210
00:19:26.160 --> 00:19:35.039
and that's data leak and prevention and and it's not equipped for non-deterministic workloads.

s211
00:19:35.039 --> 00:19:46.720
I think there's something completely different about these and you can't just use strings and you can't just try to reason in a small box about what the agent's doing.

s212
00:19:46.720 --> 00:19:57.039
So, uh, one of the things that I really like to experiment with is how do I hook the agent at runtime with a set of policies, not trying to detect,

s213
00:19:57.039 --> 00:20:00.400
you know, on the output, but on the input, giving it the right guard rails.

s214
00:20:00.400 --> 00:20:01.679
And I I I like that.

s215
00:20:01.679 --> 00:20:04.559
I like that approach a lot.

s216
00:20:09.440 --> 00:20:12.320
Uh, sort of build building on that first.

s217
00:20:12.320 --> 00:20:12.799
Thank you.

s218
00:20:12.799 --> 00:20:13.840
This is very very cool.

s219
00:20:13.840 --> 00:20:19.840
Um wondering so the ideas here completely aligned with where where do you see this existing?

s220
00:20:19.840 --> 00:20:21.440
Is this at the tool call level?

s221
00:20:21.440 --> 00:20:29.760
Is this every single turn it runs through this sort of process like how how how might you actually instrument this in practice?

s222
00:20:29.760 --> 00:20:34.000
I I I think this has to be instrumented in the harness.

s223
00:20:34.000 --> 00:20:39.039
I am not a deep enough engineer to know how that would work.

s224
00:20:39.039 --> 00:20:45.039
This is this is my plea to you all who are way more intelligent about this than I am.

s225
00:20:45.039 --> 00:20:54.559
But what I what I've seen kind of same answer I gave before like what I've seen in the things that we've built is when we can intercept an agent

s226
00:20:54.559 --> 00:21:02.880
that's about to write a line of code and say, "Hey, by the way, here's our standard for authentication.

s227
00:21:02.880 --> 00:21:07.600
Make sure you use that library right at that time before it writes the line.

s228
00:21:07.600 --> 00:21:08.240
It works."

s229
00:21:08.240 --> 00:21:21.919
So I think the question is like what do you do as a post tool hook and is that the right place to do that pro probably but again I'm out of my depth at that point

s230
00:21:21.919 --> 00:21:24.240
is that okay

s231
00:21:24.240 --> 00:21:26.320
all right

s232
00:21:39.534 --> 00:21:41.534
[music]