WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=MkRYPFIMCSA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/MkRYPFIMCSA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.280 --> 00:00:15.759
How's it going?

s3
00:00:16.240 --> 00:00:18.640
Um, my name is Ryan.

s4
00:00:18.640 --> 00:00:29.519
Um, I'm going to I I'm I'm the CEO at Dino and uh yeah, you been developing software for for quite a while at this point.

s5
00:00:29.519 --> 00:00:32.640
You might know one of my projects, Node.js.

s6
00:00:32.640 --> 00:00:39.120
Um, I want to talk about um a service that we're running at Dino called Dino Deploy.

s7
00:00:39.120 --> 00:00:45.840
This is a system for hosting websites and it has incidences.

s8
00:00:45.840 --> 00:00:51.680
It's it it has downtime occasionally and uh we've got a pager duty that fires.

s9
00:00:51.680 --> 00:00:58.000
I'm sure you're all very familiar with the very scary alarm sound that wakes you up in the middle of the night.

s10
00:00:58.000 --> 00:01:06.880
Um, and recently we've been playing around with using agents to automatically service these incidences.

s11
00:01:06.880 --> 00:01:12.080
Um, in particular, OpenClaw, but other other agents as well.

s12
00:01:12.080 --> 00:01:17.840
Um, and we've found a pattern that is working pretty well for us that I want to share with you.

s13
00:01:17.840 --> 00:01:23.360
Um, we actually give OpenClaw access to all sorts of systems.

s14
00:01:23.360 --> 00:01:29.600
Postgress, Kubernetes, Clickhouse, AWS, GitHub, Slack, uh all all sorts of things.

s15
00:01:29.600 --> 00:01:35.920
And we we do actually give them uh rewrite access to these systems.

s16
00:01:36.000 --> 00:01:44.240
This is very powerful uh because the agents can actually get all of the context.

s17
00:01:44.240 --> 00:01:46.880
They can see traces in ClickHouse.

s18
00:01:46.880 --> 00:01:54.240
They can look in the production Postgress database at what the user what you what projects a user owns.

s19
00:01:54.240 --> 00:02:05.119
They they can look through Slack for uh communications uh GitHub logs etc. Um this actually works quite well.

s20
00:02:05.119 --> 00:02:16.640
Uh the the agents are actually able to solve quite a lot of incidences where we previously would have a human s in the loop.

s21
00:02:16.640 --> 00:02:25.120
But it is very dangerous of course because these agents could do nefarious things.

s22
00:02:25.120 --> 00:02:32.000
They could start a psql subprocess and issue a delete users table.

s23
00:02:32.000 --> 00:02:36.080
Um they could call cubecuddle delete namespace prod.

s24
00:02:36.080 --> 00:02:42.080
Um you know they they could decide somehow that solving the incident it means you know removing all of the users.

s25
00:02:42.080 --> 00:02:45.519
Uh and of course we don't want that.

s26
00:02:47.200 --> 00:02:52.160
We use Opus and Opus is remarkably well aligned.

s27
00:02:52.160 --> 00:03:01.519
You can really not you you can try very hard to to get it to delete the user's table and it will refuse over and over again.

s28
00:03:01.519 --> 00:03:06.239
But this is not sufficient, right?

s29
00:03:06.239 --> 00:03:12.879
Security can't just be wishful thinking that Opus will always obey your your wishes.

s30
00:03:12.879 --> 00:03:28.800
Um these S sur agents that we have are connected to the support system and thus can be prompt injected from the outside and that means that they can be manipulated

s31
00:03:28.800 --> 00:03:29.519
somehow.

s32
00:03:29.519 --> 00:03:43.120
like who knows who knows what sort of uh string of characters could send opus into some uh bad state that allows it to think that it's taking the right action by doing something very

s33
00:03:43.120 --> 00:03:44.720
undesirable.

s34
00:03:44.720 --> 00:03:52.239
So you know we take the stance that the sec the agents themselves have to be untrusted software.

s35
00:03:52.239 --> 00:03:56.720
You can't rely on the agent itself to guard what it's doing.

s36
00:03:56.720 --> 00:04:00.239
You can't put the guard inside the agent.

s37
00:04:01.200 --> 00:04:09.120
We run agents and I assume many of you do the same on standalone VMs.

s38
00:04:09.120 --> 00:04:15.280
So we're not very concerned about agents touching files on the file system.

s39
00:04:15.280 --> 00:04:22.400
You know, they're they're they're uh they're properly isolated at the system level.

s40
00:04:22.960 --> 00:04:36.560
But [clears throat] so you know effectively every nefarious action that that an agent could take every good action that it takes comes in the form of some network communication

s41
00:04:36.560 --> 00:04:44.560
some some bites over the wire and how these bytes are formed can happen in various ways.

s42
00:04:44.560 --> 00:05:03.759
You can of course call through MCP but also subprocesses and if you think of Postgress for example this is a nonHTTP protocol that uh open clock can just spawn as as a subprocess and and start connecting to to services.

s43
00:05:03.759 --> 00:05:13.680
Um so we take the stance that we really want to understand what the bites are coming out of that agent in great detail.

s44
00:05:14.403 --> 00:05:14.960
[clears throat]

s45
00:05:14.960 --> 00:05:18.479
This can get very tricky in real world systems.

s46
00:05:18.479 --> 00:05:33.199
So for example, we have a production Postgress database in AWS um that is inside a VPC that we can only reach really through uh an EKS endpoint.

s47
00:05:33.199 --> 00:05:46.800
And what we'd really like to do is ensure that our agent, which we want to give access to everything essentially, can't somehow tunnel through this EKS server, spawn psql,

s48
00:05:46.800 --> 00:05:49.360
and drop the users table, right?

s49
00:05:49.360 --> 00:05:57.120
We're we're we're concerned about pretty crazy situations like this that get very complicated.

s50
00:05:57.120 --> 00:06:04.960
And I think many of you work in companies where you have real world systems where things are very complic.

s51
00:06:06.479 --> 00:06:19.199
So yeah, just to just to highlight this, this is an outbound path the agents host can't reach on a protocol that isn't HTTP that's gated by a rule that understands SQL.

s52
00:06:19.919 --> 00:06:23.680
These are what human S surres would do.

s53
00:06:23.680 --> 00:06:34.560
And how can we, you know, empower these these agents to to have kind of the the same access that that a human might.

s54
00:06:34.560 --> 00:06:46.080
Um, so you might ask, you might say, well, you know, there's ACL's, there's permissions, you can issue readonly uh Postgress credentials.

s55
00:06:46.080 --> 00:06:49.280
Um, and yeah, that's true up to a point.

s56
00:06:49.280 --> 00:06:53.680
Um, you can do careful credential provisioning and you should.

s57
00:06:53.680 --> 00:07:02.560
Um, but this this really requires uh working across many different systems, provisioning credentials in in incredibly careful ways.

s58
00:07:02.560 --> 00:07:10.880
And as I just demonstrated, the composition of access can lead to holes when you can access one system and then another system.

s59
00:07:10.880 --> 00:07:24.800
um MCP, you know, you can you can uh structure all of this as uh very careful MCP uh tools that uh have the proper permissions.

s60
00:07:24.800 --> 00:07:28.080
But, you know, then then you can't spawn subprocesses, right?

s61
00:07:28.080 --> 00:07:39.440
You you you can't you know, as soon as as soon as the open clause spawns the the PSQL, uh you're you're kind of out broken through the the security boundary.

s62
00:07:40.639 --> 00:07:55.520
There are quite a few projects in this space um namely projects that kind of sit in front of an agent and under look at what it's sending and try to control based on on

s63
00:07:55.520 --> 00:07:59.280
uh the the bytes that are flowing through this.

s64
00:07:59.280 --> 00:08:00.560
Um, LLM gateways.

s65
00:08:00.560 --> 00:08:04.479
I think we're all familiar with Open Router, Light LLM, for example.

s66
00:08:04.479 --> 00:08:19.840
These often have a guard rails feature that can [clears throat] uh guard against prompt injection uh, you know, scan for for various uh, expressions, etc. that that are going back and forth between

s67
00:08:19.840 --> 00:08:24.560
the the LLM uh, provider.

s68
00:08:24.800 --> 00:08:26.879
But, of course, that's just the LLM.

s69
00:08:26.879 --> 00:08:30.720
uh you know we're we're talking to databases and stuff.

s70
00:08:30.720 --> 00:08:47.120
Um you have systems like HTTP jail and Crabtrap that are HTTP proxies that really sit at at the HTTP layer and you uh HTTP jail for example can will allow you to write rules that say

s71
00:08:47.120 --> 00:08:53.120
well you can make get requests but not post requests or you can access this HTTP subpath.

s72
00:08:53.120 --> 00:09:03.920
Um, Crabt Trap is a project from Brex that has a LLM as judge that operates on the HTTP requests flowing back and forth.

s73
00:09:03.920 --> 00:09:12.399
you have uh proxies that inject credentials into uh as they're passing out of the agent.

s74
00:09:12.399 --> 00:09:27.600
Uh agent vault uh being a popular one where the the agent itself never actually sees the credentials of the system that it's talking to but passes some placeholder out and the proxy itself injects those credentials.

s75
00:09:27.600 --> 00:09:32.399
This is an important part of the problem but not a complete solution.

s76
00:09:32.399 --> 00:09:46.640
And you have things like process sandboxes like Nvidia's OpenShell that you know really are kind of OS system level uh uh guards against say accessing different file system paths

s77
00:09:46.640 --> 00:10:00.000
um accessing different SIS calls that sort of thing but as I said before we're we're not really concerned about that because we provision uh standalone VM for for our agents.

s78
00:10:00.320 --> 00:10:07.680
So the software that uh we've written to address this problem is called claw patrol.

s79
00:10:07.680 --> 00:10:16.880
Uh it's an open- source MIT license project and this is a proxy that sits in front of your agents.

s80
00:10:16.880 --> 00:10:22.480
Um it operates not at the HTP level but at a lower level.

s81
00:10:22.480 --> 00:10:28.320
It understands each and every bite flowing through flowing out of your agent.

s82
00:10:28.320 --> 00:10:42.560
It holds credentials like agent vault and can inject those credentials so that your uh whatever agent software you're using uh doesn't actually doesn't ever actually see secret values.

s83
00:10:42.560 --> 00:11:02.560
And um in particular, it has a very advanced rule system that allows you to say in in precise details how how and and what requests get uh transferred out out of the agent and talk to the outside world.

s84
00:11:02.560 --> 00:11:13.040
These rules are are kind of the the key piece of the system and we write them in a configuration file using a language called HCL.

s85
00:11:13.040 --> 00:11:15.839
Uh who anybody familiar with HCL?

s86
00:11:15.839 --> 00:11:19.440
This is like the Terraform the Terraform configuration language.

s87
00:11:19.440 --> 00:11:21.279
Uh it actually works really well here.

s88
00:11:21.279 --> 00:11:34.720
So we have a file that we check into git and we manage very carefully that essentially defines the permissions for all of our services at Dino and these yeah it's it's a big long file.

s89
00:11:34.720 --> 00:11:41.519
It's like a thousand lines and you know we we manage each and every change to that in in kind of precise detail.

s90
00:11:41.519 --> 00:11:52.000
This is an example of a rule in our configuration file that blocks certain Postgress functions from being uh being called.

s91
00:11:52.000 --> 00:12:02.959
And so yeah, again Postgress being a nonHTP protocol and these rules can be applied even when tunneling through other systems.

s92
00:12:02.959 --> 00:12:16.639
Um, it supports uh a number of different protocols and has a plug-in system to extend it when you run into a protocol that it is not yet familiar with.

s93
00:12:16.720 --> 00:12:18.880
So, uh, here's here's a little demo.

s94
00:12:18.880 --> 00:12:20.240
Unfortunately, not live.

s95
00:12:20.240 --> 00:12:29.360
Um, but we call claw patrol run codeex in yellow mode so that it just does what you say it should do.

s96
00:12:29.360 --> 00:12:44.720
And you tell you tell Codex, hey, delete the users table from from Postgress and Codeex um properly uh obeys and starts a Psql subprocess where it deletes the the the

s97
00:12:44.720 --> 00:12:45.839
users table.

s98
00:12:45.839 --> 00:12:58.399
That uh Psql subprocess opens a network connection to to our to the the Postgress server that goes through claw patrol where we pars each and every bite.

s99
00:12:58.399 --> 00:13:01.440
We understand the Postgress protocol.

s100
00:13:01.440 --> 00:13:12.399
We apply our rules and ultimately reject that what we call an action from uh from doing something destructive.

s101
00:13:14.480 --> 00:13:20.720
Claw Patrol has a dashboard that lets you see what your agents are doing.

s102
00:13:20.720 --> 00:13:29.279
So at the top you can see a couple of different uh devices or agents and the the various requests that are flowing through.

s103
00:13:29.279 --> 00:13:35.680
Some of them being denied, some of them need approval which I'll talk about in a second.

s104
00:13:35.680 --> 00:13:47.360
And you can click into to each request or uh action as we call it because it's more general than HTTP requests and see the details of of what's going on.

s105
00:13:47.360 --> 00:13:52.160
there's there's analytics and yeah it's it's very uh utilitarian driven.

s106
00:13:52.160 --> 00:13:56.720
It's like what what we need to understand our own agents.

s107
00:13:56.720 --> 00:14:01.360
Um there as I said there's there's an approval system in this.

s108
00:14:01.360 --> 00:14:13.279
So you can route, you can define rules that don't just reject requests or actions but uh ask a human for example in a slack channel or run an LLM

s109
00:14:13.279 --> 00:14:18.560
judge over over this um or any combination thereof, right?

s110
00:14:18.560 --> 00:14:23.760
Maybe first first get an LLM judge and then get approval in Slack.

s111
00:14:23.760 --> 00:14:34.639
uh so that you can have again very precise control over what your agents are doing outside of the agent software itself right we we treat the agent software as

s112
00:14:34.639 --> 00:14:54.720
a black box right we we don't require any changes to to that software um I mentioned credential injection before uh claw patrol has very detailed support for all sorts of systems credentials come in many different forms they're not just

s113
00:14:54.720 --> 00:15:09.360
uh bearer uh headers uh it handles cookies it handles Postgress as I mentioned uh click house supports all sorts of uh ooth protocols supports very complex things like AWS

s114
00:15:09.360 --> 00:15:24.639
SIG v4 um so yeah this I guess what I'm trying to uh say is that this is this is really born out of utility here and meant for real world systems This is not just

s115
00:15:24.639 --> 00:15:28.399
you know kind of an imaginary scenario.

s116
00:15:28.800 --> 00:15:35.519
[clears throat] Um this system works over tail scale or wire guard.

s117
00:15:35.519 --> 00:15:46.880
Um we ourselves run claw patrol run our agents inside of tail scale inside of a tail net and claw patrol acts as a tail scale exit node.

s118
00:15:46.880 --> 00:15:52.320
Um we also lean on tail scale for authentication to the dashboard.

s119
00:15:52.320 --> 00:16:04.320
So your your tail your tail scale identity actually allows you access to to the dashboard so that we don't have to layer on another authentication mechanism.

s120
00:16:04.320 --> 00:16:11.199
But we also have this wire guard for people who have not bought into the wonderful tail scale ecosystem.

s121
00:16:11.199 --> 00:16:23.440
But this works very well for us because we know that all of our stuff is is off the internet and all of these very security sensitive things are are uh tightly controlled.

s122
00:16:23.440 --> 00:16:27.759
Claw Patrol itself is holding all of these credentials to production systems.

s123
00:16:27.759 --> 00:16:31.920
So you have to be very careful with it.

s124
00:16:32.160 --> 00:16:39.680
Um so yeah, this the thesis here is is basically that agents can't be trusted to police themselves.

s125
00:16:39.680 --> 00:16:46.560
that includes security plugins or or modifications to the to the agent software itself.

s126
00:16:46.560 --> 00:16:49.360
The the security boundary has to be elsewhere.

s127
00:16:49.360 --> 00:17:00.480
And that's not to say that alignment is not a good thing, but uh you know for for real world security systems, we we really do need to control this at at a higher level.

s128
00:17:00.480 --> 00:17:06.400
And uh claw patrol is our attempt to uh make this work for ourselves.

s129
00:17:06.400 --> 00:17:10.559
Um and yeah, you can you can check it out here.

s130
00:17:11.636 --> 00:17:13.636
[applause]

s131
00:17:17.600 --> 00:17:22.880
I might have time for one question or so.

s132
00:17:22.880 --> 00:17:23.520
Yes sir.

s133
00:17:23.520 --> 00:17:29.039
What kind of email testing do you do on this to make sure it's working?

s134
00:17:29.039 --> 00:17:29.679
Uh yeah.

s135
00:17:29.679 --> 00:17:34.559
So the question is what what sort of testing do we do we do to make make sure it works properly?

s136
00:17:34.559 --> 00:17:48.640
Um I I didn't mention but but there this rule file actually has a test system along with it where you can provide uh fixtures action like fixture requests that can flow through the rules and then you can

s137
00:17:48.640 --> 00:17:57.120
uh essentially create unit tests to make sure that that fixture is always you know that request it will always be blocked by by your set of rules.

s138
00:17:57.120 --> 00:18:04.240
And then of course for the claw patrol software itself we have a a large suite of of testing.

s139
00:18:04.559 --> 00:18:06.799
Yes sir.

s140
00:18:10.480 --> 00:18:16.080
So the question is as as agents get smarter does this problem get bigger or smaller?

s141
00:18:16.080 --> 00:18:22.160
I think I think we can we will never be able to fully trust uh AIs.

s142
00:18:22.160 --> 00:18:32.320
I think it becomes less and less of a problem as they are smarter, have better context, know that they're working with a company, know that that they shouldn't be doing bad things.

s143
00:18:32.320 --> 00:18:42.160
Opus is more aligned than previous models, but I think we're always going to have to have uh backs stop security mechanisms.

s144
00:18:42.160 --> 00:18:43.039
Um, cool.

s145
00:18:43.039 --> 00:18:46.612
Well, I I'll be around for other questions, but thank you very much.

s146
00:18:46.612 --> 00:18:48.612
[applause]

s147
00:19:04.298 --> 00:19:06.298
[music]