WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=26RtyAm9y_Q

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/26RtyAm9y_Q.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.599 --> 00:00:15.120
All right.

s3
00:00:15.120 --> 00:00:18.640
The dark arts of web automation.

s4
00:00:18.640 --> 00:00:20.880
That sounds ominous, right?

s5
00:00:20.880 --> 00:00:25.039
Sounds like something I'm going to teach you that you're going to need a lawyer for.

s6
00:00:25.039 --> 00:00:29.439
Uh, well, actually, we'll come back to the lawyer in a minute, but a little bit of background.

s7
00:00:29.439 --> 00:00:39.280
As I was preparing for this talk, OpenAI threatened to ban my account just for the work that I was doing in preparing for the talk.

s8
00:00:42.879 --> 00:00:50.239
So I checked my inbox a few days ago and got this and that's a real shocker, right?

s9
00:00:50.239 --> 00:00:59.360
So what does one do to earn the banhammer for cyber abuse with a web browser?

s10
00:00:59.760 --> 00:01:01.440
this.

s11
00:01:01.440 --> 00:01:03.440
I was doing this.

s12
00:01:03.440 --> 00:01:12.240
So, what you see here is every one of these is being solved by an AI agent with no human in the loop.

s13
00:01:12.240 --> 00:01:18.159
And in 15 minutes or so, you'll understand exactly how this is possible.

s14
00:01:18.159 --> 00:01:22.000
I want my agents to be able to use the web exactly the way that I do.

s15
00:01:22.000 --> 00:01:29.920
I want them to book the things, to send the emails, to fill in the forms so that I don't have to.

s16
00:01:29.920 --> 00:01:37.280
But the moment that something isn't a person starts clicking into a page, web pages often fight back.

s17
00:01:37.280 --> 00:01:40.159
So this talk is about winning that fight.

s18
00:01:40.159 --> 00:01:44.880
And it starts with one slightly unconventional idea.

s19
00:01:45.119 --> 00:01:46.079
And here it is.

s20
00:01:46.079 --> 00:01:49.920
This is the premise of the whole talk on one slide.

s21
00:01:49.920 --> 00:01:56.240
A CDP browser is just like a meatag with a mouse.

s22
00:01:56.399 --> 00:01:57.040
No joke.

s23
00:01:57.040 --> 00:02:01.520
Well, at least as far as Google and Cloudflare and the rest can tell.

s24
00:02:01.520 --> 00:02:02.079
No joke.

s25
00:02:02.079 --> 00:02:07.726
If you have a browser, an agent drive a browser using the Chrome DevTools protocol,

s26
00:02:07.726 --> 00:02:08.080
[snorts]

s27
00:02:08.080 --> 00:02:15.280
your agents clicks and keystrokes travel the exact same path inside Chrome that yours do.

s28
00:02:15.280 --> 00:02:16.560
And that's the big idea.

s29
00:02:16.560 --> 00:02:19.440
The rest of the talk is about how to pull it off.

s30
00:02:19.440 --> 00:02:33.760
And that comes down to three things to you need a a CLI, not an MCP, the Chrome DevTools protocol, which is where a tool that I wrote called Chrome Agent comes into play,

s31
00:02:33.760 --> 00:02:37.120
and then a loop run on a ladder.

s32
00:02:37.120 --> 00:02:40.239
So, let's start with the one that picks a fight.

s33
00:02:40.239 --> 00:02:45.680
The first thing is to give your agent a CLI and not an MCP server.

s34
00:02:45.680 --> 00:02:48.720
And by CLI, I mean shell-based tools.

s35
00:02:48.720 --> 00:02:56.239
Before you in the back knuckle up on this one, there are some specific reasons as to why this is necessary.

s36
00:02:56.239 --> 00:02:58.959
First of all, it's worth noting that capability is a wash.

s37
00:02:58.959 --> 00:03:10.480
So, in a recent study by the guys at Arise AI, both a CLI and an MCP given the same task achieve those tasks successfully roughly 83% of the time.

s38
00:03:10.480 --> 00:03:18.400
However, a CLI beats MCP in reuse, in speed, and in cost.

s39
00:03:18.400 --> 00:03:20.319
So, let's take reuse first.

s40
00:03:20.319 --> 00:03:23.120
A CLI sequence can be programmed.

s41
00:03:23.120 --> 00:03:32.400
You write it once and you run it a thousand times without a model in the loop, whereas MCP hits the model on every single turn.

s42
00:03:32.400 --> 00:03:37.680
The CLI is faster for a similar reason, because there's not a model in the middle of every step.

s43
00:03:37.680 --> 00:03:50.959
So in that same study that I mentioned before, MCP took 71 round trips and 8 minutes for the same task that took a CLI only seven turns and in under one minute.

s44
00:03:50.959 --> 00:03:55.599
So hold on to speed because that actually comes back at the end.

s45
00:03:55.599 --> 00:03:57.040
Lastly, token cost.

s46
00:03:57.040 --> 00:04:06.159
Anthropic themselves reported that the CLI can be as much as 75 times cheaper in terms of token cost.

s47
00:04:06.159 --> 00:04:10.000
So what is it that we're actually running on the command line?

s48
00:04:10.000 --> 00:04:14.239
We drive the browser in the Chrome DevTools protocol.

s49
00:04:14.239 --> 00:04:17.519
And that's the second thing that your agent needs to appear human.

s50
00:04:17.519 --> 00:04:20.000
You already know this protocol even if you don't know its name.

s51
00:04:20.000 --> 00:04:26.320
That panel that pops up when you hit F12 that drives the browser using exactly this protocol.

s52
00:04:26.320 --> 00:04:28.400
Your agents can speak it too.

s53
00:04:28.400 --> 00:04:30.320
They can use the Chrome agent tool.

s54
00:04:30.320 --> 00:04:37.723
And Chrome Agent also makes it really easy for your agents to write code to replay CDP interactions.

s55
00:04:37.723 --> 00:04:37.759
[snorts]

s56
00:04:37.759 --> 00:04:42.880
The surface area of CDP is enormous and it changes really frequently.

s57
00:04:42.880 --> 00:04:49.759
So as of now it's 57 domains and within those there are hundreds and hundreds of methods and events.

s58
00:04:49.759 --> 00:04:55.680
So I've kind of bucketed all of the domains into eight buckets to make it a little easier to keep it in your head.

s59
00:04:55.680 --> 00:04:59.440
But no worries, you don't actually need all 57 domains.

s60
00:04:59.440 --> 00:05:04.880
In order to interact with a browser in the way that a human does, you need a small subset of these.

s61
00:05:04.880 --> 00:05:12.639
And the best way to think about that small subset is in terms of the digital senses that they give your agent.

s62
00:05:12.639 --> 00:05:23.840
So you see the page, you may read its structure from the DOM, you may read its semantics from the accessibility tree, or just take a screenshot if you need the pixels.

s63
00:05:23.840 --> 00:05:25.680
you hear the page.

s64
00:05:25.680 --> 00:05:39.360
So that may be network traffic to and from uh or the console or logs and then you operate the page with clicks and with keystrokes and with navigation.

s65
00:05:39.440 --> 00:05:47.360
The third and final thing that you need for your agent to appear human is what I like to call a loop on a ladder.

s66
00:05:47.360 --> 00:05:48.880
So here's the loop.

s67
00:05:48.880 --> 00:05:55.120
sense, act, verify, and you repeat that until the page gives in.

s68
00:05:55.120 --> 00:05:59.520
So in sensing, we perceive the page through one or more channels.

s69
00:05:59.520 --> 00:06:05.600
The DOM, the accessibility tree, a screenshot as I just mentioned, and then you act, you do one thing.

s70
00:06:05.600 --> 00:06:12.479
You click something, you type something, you take a you uh select something, and then you verify.

s71
00:06:12.479 --> 00:06:17.680
And now this is sensing again, but it's sensing through a different channel than the interaction.

s72
00:06:17.680 --> 00:06:23.440
So, for example, if you've clicked something, don't ask the click if it was successful.

s73
00:06:23.440 --> 00:06:26.479
Check the network or check the screen.

s74
00:06:26.479 --> 00:06:28.240
So, you sense where you are.

s75
00:06:28.240 --> 00:06:33.039
You make one move, you confirm that it landed, and then you iterate again.

s76
00:06:33.039 --> 00:06:40.000
And when the loop won't close, so you sense, act, verify, and the page still won't do what it is you're trying to do.

s77
00:06:40.000 --> 00:06:41.759
That's the page fighting back.

s78
00:06:41.759 --> 00:06:46.319
And so, that's telling you to climb the ladder, the meatag ladder.

s79
00:06:46.319 --> 00:06:53.120
That is this is a ladder of techniques that are increasingly more human as you climb it.

s80
00:06:53.120 --> 00:06:57.680
It has three rungs and you climb only as high as the page forces you.

s81
00:06:57.680 --> 00:07:03.599
Or said differently, you you climb to the lowest ladder that the lowest rung, sorry, that actually works.

s82
00:07:03.599 --> 00:07:03.789
So

s83
00:07:03.789 --> 00:07:03.840
[snorts]

s84
00:07:03.840 --> 00:07:06.479
on rung one, you don't act human at all.

s85
00:07:06.479 --> 00:07:15.039
If you can just use the API that's exposed within the page then and issue a synthetic JavaScript click, then do that.

s86
00:07:15.039 --> 00:07:16.319
It's easy.

s87
00:07:16.319 --> 00:07:17.280
It's free.

s88
00:07:17.280 --> 00:07:18.080
It's instant.

s89
00:07:18.080 --> 00:07:20.319
And it's the right default.

s90
00:07:20.319 --> 00:07:25.120
You climb to rung two when faking it stops working.

s91
00:07:25.120 --> 00:07:31.120
So when you need to use a real click, for example, using the CDP input domain.

s92
00:07:31.120 --> 00:07:37.039
Now this is agent input that the page cannot tell apart from your own.

s93
00:07:37.039 --> 00:07:42.560
You climb to rung three when you need human input and human behavior.

s94
00:07:42.560 --> 00:07:52.000
So this is a real mouse path maybe with a little dwell and some jitter uh or vision to actually see and interpret things.

s95
00:07:52.000 --> 00:08:02.560
So you start cheap, you climb run one rung rung at a time only when the page makes you climb the ladder loop on each rung and then write down

s96
00:08:02.560 --> 00:08:04.720
the path that worked.

s97
00:08:04.720 --> 00:08:10.240
And that's how you arrive at fully automated AI agent-driven browsing.

s98
00:08:10.240 --> 00:08:13.440
First you explore You run the loop by hand.

s99
00:08:13.440 --> 00:08:16.800
You climb rungs until you the thing actually works.

s100
00:08:16.800 --> 00:08:18.319
And then you automate.

s101
00:08:18.319 --> 00:08:22.720
You write the solution down so you never have to figure it out again.

s102
00:08:22.720 --> 00:08:29.520
And you write it down as code or as an agent skill or very often as both.

s103
00:08:29.520 --> 00:08:31.840
So let me show you how this all comes together.

s104
00:08:31.840 --> 00:08:35.039
But first of all with a word from my attorney.

s105
00:08:35.039 --> 00:08:43.839
So, I mentioned that the lawyer would be back and everything you're about to see is in fact real agent browser use in the wild.

s106
00:08:43.839 --> 00:08:45.519
It's taken from real experience.

s107
00:08:45.519 --> 00:08:55.120
However, upon the advice of council, everything that you're seeing is running only on infrastructure that I own and accounts that I operate right now.

s108
00:08:55.120 --> 00:09:00.240
So, who's ready to see the agents get busy with the browser?

s109
00:09:01.200 --> 00:09:02.880
All right.

s110
00:09:02.880 --> 00:09:06.880
So, here's a simple one, everyday use case we'll start with.

s111
00:09:06.880 --> 00:09:13.440
Let's send a batch of personalized emails, each one different from your Outlook web client.

s112
00:09:13.440 --> 00:09:16.320
And we've got some lovely pseudo code on the right side.

s113
00:09:16.320 --> 00:09:17.120
Oh, sorry, left side.

s114
00:09:17.120 --> 00:09:17.839
It's inverted.

s115
00:09:17.839 --> 00:09:24.959
Left side of the uh the slide that makes it really simple for you to see how it all kind of works in code.

s116
00:09:24.959 --> 00:09:27.839
Outlook's compose box has nothing for us to defeat.

s117
00:09:27.839 --> 00:09:30.560
So, this is rung one of the meatag ladder.

s118
00:09:30.560 --> 00:09:32.560
You don't act human at all.

s119
00:09:32.560 --> 00:09:35.279
A synthetic click opens the compose window.

s120
00:09:35.279 --> 00:09:41.519
We fill this in programmatically and another synthetic click sends it and then you let that rip.

s121
00:09:41.519 --> 00:09:41.920
Right?

s122
00:09:41.920 --> 00:09:46.720
So the reason this is going so smoothly is it's executing a program.

s123
00:09:46.720 --> 00:09:56.720
So you capture the sequence once and then loop it whether it's 20 emails or 200 and from from just that one command.

s124
00:09:56.720 --> 00:10:00.720
So you solve it once and reuse it forever.

s125
00:10:00.720 --> 00:10:08.880
The agent is riffing on content to personalize here as we go, but it's running a program to drive all of the interactions.

s126
00:10:08.880 --> 00:10:13.519
Now, you may ask, why would we drive the web UI at all?

s127
00:10:13.519 --> 00:10:16.720
Like, why not just use the API?

s128
00:10:16.720 --> 00:10:29.040
And in corporate environments, the API for an Office 365 tenant requires an app registration and it also requires admin approval, which as an employee, you can't often get.

s129
00:10:29.040 --> 00:10:36.320
And so in this pattern or sorry whereas the the web login you have is all you need to be able to do exactly what you see on the screen here now.

s130
00:10:36.320 --> 00:10:45.519
And so in this pattern the web UI itself kind of becomes a universal API right like a permissionless API which is really neat.

s131
00:10:45.519 --> 00:10:51.600
So that's rung one but what happens when the page starts fighting back?

s132
00:10:51.920 --> 00:10:59.600
So, let's say that you're browsing on your favorite online mega store and we'll just call them Demon.

s133
00:10:59.680 --> 00:11:05.200
So, they're a crafty bunch over there at Amazon and they have no love for your bots at all.

s134
00:11:05.200 --> 00:11:14.959
Now, if you take that same JavaScript click that just worked in Outlook and you point it at say the add to cart button, you get nothing.

s135
00:11:14.959 --> 00:11:15.760
Literally nothing.

s136
00:11:15.760 --> 00:11:17.040
There's there's no failure.

s137
00:11:17.040 --> 00:11:18.079
There's no error.

s138
00:11:18.079 --> 00:11:22.000
just nothing happens and the page is ignoring it.

s139
00:11:22.000 --> 00:11:29.760
And the reason for that is the page is checking was this click from a human source.

s140
00:11:29.760 --> 00:11:35.519
Chrome stamps every single event with just that answer whether it's trusted or untrusted.

s141
00:11:35.519 --> 00:11:40.959
So the JavaScript click that we fired previously is stamped untrusted.

s142
00:11:40.959 --> 00:11:45.360
And in this case, the page just quietly drops that input.

s143
00:11:45.360 --> 00:11:46.240
But that's no worries.

s144
00:11:46.240 --> 00:11:47.839
we'll just climb the meatag ladder.

s145
00:11:47.839 --> 00:11:57.760
So, we move to rung two and we click using Chrome's input domain and that uses the exact same input path that your actual mouse uses.

s146
00:11:57.760 --> 00:12:08.240
And now it's stamped trusted and the page can't tell the difference between your mouse and our agent and bam, the items drop straight into the cart.

s147
00:12:08.240 --> 00:12:11.440
So, we've got quite the inside view of demazon.com here.

s148
00:12:11.440 --> 00:12:17.120
So if you look at this lower panel, you can imagine that this is what the pages logs look like.

s149
00:12:17.120 --> 00:12:22.800
Every one of the untrusted clicks fails, but the trusted ones go through.

s150
00:12:22.800 --> 00:12:24.320
No worries.

s151
00:12:24.320 --> 00:12:38.639
Now, as a heads up going forward, when you see this mouse cursor moving here, that is added programmatically just so that you can visually see the mouse inputs that the agent is giving to the browser using Chrome agent and CDP.

s152
00:12:38.639 --> 00:12:41.600
It's not actually moving my mouse here.

s153
00:12:41.600 --> 00:12:46.560
All right, rung two is where the real meatag inputs begin.

s154
00:12:46.560 --> 00:12:50.800
But that's not enough to replace you in the browser.

s155
00:12:50.800 --> 00:12:54.639
So we climb to the top of the meatag ladder, rung three.

s156
00:12:54.639 --> 00:12:59.680
And this is the narrow frontier where pages are actively hunting for bots.

s157
00:12:59.680 --> 00:13:02.320
There are a variety of techniques that we're going to deploy here though.

s158
00:13:02.320 --> 00:13:04.880
So let's talk through a few of them.

s159
00:13:04.880 --> 00:13:06.959
I'm sure this guy looks familiar, right?

s160
00:13:06.959 --> 00:13:10.079
This is Cloudflare turn style.

s161
00:13:10.079 --> 00:13:15.120
It looks deceptively simple, but it is the hardest target that we've hit yet.

s162
00:13:15.120 --> 00:13:28.959
And it's because that little checkbox can't be easily reached through typical web automation programming because these cheeky guys have hidden this thing through isolated boundaries.

s163
00:13:28.959 --> 00:13:33.279
First, it's encapsulated beneath a closed shadow route.

s164
00:13:33.279 --> 00:13:41.040
And then the whole widget itself lives in a cross origin iframe which in it then also has another shadow route.

s165
00:13:41.040 --> 00:13:46.720
So to every cheap trick that that checkbox is uncclickable.

s166
00:13:46.720 --> 00:13:48.959
There's no element to grab.

s167
00:13:48.959 --> 00:13:50.079
So what do we do?

s168
00:13:50.079 --> 00:13:52.560
Well, we just stop trying to grab it, right?

s169
00:13:52.560 --> 00:13:57.760
We ask the browser where it is that the iframe sits on the screen.

s170
00:13:57.760 --> 00:14:01.120
We do a little bit of math to figure out where the checkbox is.

s171
00:14:01.120 --> 00:14:08.320
Then we fire a trusted click right at that position on the glass and then Chrome does the rest for us, right?

s172
00:14:08.320 --> 00:14:12.480
A real click lands right in the checkbox and we're off and running.

s173
00:14:12.480 --> 00:14:13.600
There's no human in the loop.

s174
00:14:13.600 --> 00:14:15.120
This is all agent.

s175
00:14:15.120 --> 00:14:17.920
So that's level one cleared.

s176
00:14:17.920 --> 00:14:21.360
And the trick here was really just kind of figuring out how to interact with it.

s177
00:14:21.360 --> 00:14:26.480
But all the next levels now make you prove that you can actually see.

s178
00:14:26.560 --> 00:14:27.839
So, this is MT capture.

s179
00:14:27.839 --> 00:14:28.800
You remember this guy?

s180
00:14:28.800 --> 00:14:30.959
And these guys are then like it.

s181
00:14:30.959 --> 00:14:32.320
They're they're still around.

s182
00:14:32.320 --> 00:14:35.519
Um, your agent's actually got to read this guy.

s183
00:14:35.519 --> 00:14:39.040
And so, what the agent does is simulate what you would do.

s184
00:14:39.040 --> 00:14:47.120
It takes a screenshot of the challenge, looks at it, and then uses its own vision capability to pick the characters out of the noise.

s185
00:14:47.120 --> 00:15:03.279
Then it types the answer back using real trusted keystrokes routed into the widgets cross origin iframe one character at a time and these are the same keyboard inputs that you would send

s186
00:15:03.279 --> 00:15:08.880
and then the server agrees the text is verified and the token is issued.

s187
00:15:08.880 --> 00:15:17.440
Now, there's one more level before the final boss, and this one is one or lost based on how it is that you move like a meat bag.

s188
00:15:17.440 --> 00:15:19.600
So, this is by Lemon.

s189
00:15:19.600 --> 00:15:25.680
Uh, it's a little jigsaw puzzle where you spot where the piece belongs and then you drag it in to fill the gap.

s190
00:15:25.680 --> 00:15:29.120
And there's an entire class of captures just like this.

s191
00:15:29.120 --> 00:15:31.920
Now, this one's Trixie in different ways.

s192
00:15:31.920 --> 00:15:33.360
There's no shadow route.

s193
00:15:33.360 --> 00:15:35.120
There's no cross origin iframe.

s194
00:15:35.120 --> 00:15:37.680
The piece is sitting right there in the page.

s195
00:15:37.680 --> 00:15:40.399
The hard part here is the drag itself.

s196
00:15:40.399 --> 00:15:49.279
So when you drop the puzzle piece, these types of captures sample the mouse movement into a trail of points the whole way.

s197
00:15:49.279 --> 00:15:53.199
And so that includes jitter and changing speeds and all of that.

s198
00:15:53.199 --> 00:15:58.480
So it's not just solving the puzzle, but it's solving it with moves like Jagger.

s199
00:15:58.480 --> 00:16:02.639
So the agent drags the exact same way that a hand would drag, right?

s200
00:16:02.639 --> 00:16:05.920
If you kind of watch him move, it eases in gently.

s201
00:16:05.920 --> 00:16:07.279
There's like a slight curve.

s202
00:16:07.279 --> 00:16:13.341
It actually deliberately overshoots the puzzle piece and then eases it right back in

s203
00:16:13.341 --> 00:16:13.680
[snorts]

s204
00:16:13.680 --> 00:16:15.600
just like a meat bag with a mouse.

s205
00:16:15.600 --> 00:16:23.839
And it's using vision to identify the gap and then humanlike motion to cross it.

s206
00:16:23.839 --> 00:16:29.040
So that's turn style, that's MT capture, and that's lemon.

s207
00:16:29.040 --> 00:16:31.519
Three gates built to keep agents out.

s208
00:16:31.519 --> 00:16:36.560
And we've just beat each one of them, which leaves only one boss standing.

s209
00:16:36.560 --> 00:16:41.120
And here he is, the final boss of the internet.

s210
00:16:41.120 --> 00:16:43.279
Recapture V2.

s211
00:16:43.279 --> 00:16:50.560
It's that little checkbox and then the blurry grid with like fuzzy pictures of crosswalks or traffic lights in it.

s212
00:16:50.560 --> 00:16:56.800
We've all squinted at these ones, but we've got the whole kit now on how to beat this guy, too.

s213
00:16:56.800 --> 00:17:06.000
The digital sensors, the loop to deploy them in, the meatag ladder, everything that you need to take down this cheeky bastard.

s214
00:17:06.000 --> 00:17:08.079
So let's go.

s215
00:17:08.079 --> 00:17:13.679
This is the whole machine and it comes in two halves.

s216
00:17:13.760 --> 00:17:15.919
On one side we have the solver.

s217
00:17:15.919 --> 00:17:17.360
This is pure code.

s218
00:17:17.360 --> 00:17:18.880
No agent, no model.

s219
00:17:18.880 --> 00:17:20.720
It does everything programmatically.

s220
00:17:20.720 --> 00:17:23.600
It does the trusted click in the checkbox.

s221
00:17:23.600 --> 00:17:34.000
It pierces into the challenge iframe and then every round it screenshots the grid and if for some reason a round expires, it just rearms itself and goes again.

s222
00:17:34.000 --> 00:17:36.400
All of that that bit there is deterministic.

s223
00:17:36.400 --> 00:17:38.492
It's fast and it's free.

s224
00:17:38.492 --> 00:17:42.559
[clears throat and cough] But there's one step in the loop that the code can't do.

s225
00:17:42.559 --> 00:17:46.480
And that's look at that grid of fuzzy tiles and figure out what it is that's in it.

s226
00:17:46.480 --> 00:17:48.640
Is this a bus for example?

s227
00:17:48.640 --> 00:17:50.240
That's vision and thinking.

s228
00:17:50.240 --> 00:17:52.880
And that needs eyes and a brain.

s229
00:17:52.880 --> 00:17:56.480
And so that's the only thing that we give the agent as a job.

s230
00:17:56.480 --> 00:17:58.720
We call that bit the operator.

s231
00:17:58.720 --> 00:18:08.320
The solver taps the agent on the shoulder and the agent takes one look at the grid, picks the tiles with whatever the thing is that we're looking for in them,

s232
00:18:08.320 --> 00:18:13.679
hands that answer back to the solver, and then just hangs out waiting until the next lab.

s233
00:18:13.679 --> 00:18:19.120
And this is really the entire talk running as one system.

s234
00:18:19.120 --> 00:18:26.480
Code does the deterministic driving and the agent does the only bits that require eyes and a brain.

s235
00:18:26.480 --> 00:18:26.720
All right.

s236
00:18:26.720 --> 00:18:29.360
Who wants to see a go?

s237
00:18:30.080 --> 00:18:32.240
There it is.

s238
00:18:32.240 --> 00:18:35.679
So, while this is playing, by the way, kind of have a look.

s239
00:18:35.679 --> 00:18:38.080
Like, some of these are really hard.

s240
00:18:38.080 --> 00:18:45.039
The the agent was able to spot bicycles in some of these tiles that I didn't see.

s241
00:18:45.039 --> 00:18:47.840
But ultimately, it's solved.

s242
00:18:47.840 --> 00:18:49.360
It's verified by the server.

s243
00:18:49.360 --> 00:18:50.799
And it's fast.

s244
00:18:50.799 --> 00:18:56.640
And fast is the whole game here because this big bad boss is on a clock.

s245
00:18:56.640 --> 00:19:03.200
Every round expires and one challenge can be multiple rounds back to back.

s246
00:19:03.200 --> 00:19:10.880
An agent that roundtrips a model on every click and on every look burns that clock and loses.

s247
00:19:10.880 --> 00:19:13.520
The challenge expires well before it ever finishes.

s248
00:19:13.520 --> 00:19:19.919
The only thing that I found that defeats this whole mess is exactly what you're looking at here.

s249
00:19:19.919 --> 00:19:25.760
deterministic code running at machine speed with one quick AI look per round.

s250
00:19:25.760 --> 00:19:29.360
So remember when I told you to hold on to speed, this is exactly why.

s251
00:19:29.360 --> 00:19:38.480
And this is also why this had to be a CLI running the CDP and not a model sitting in the middle of every single interaction.

s252
00:19:38.480 --> 00:19:41.440
By the way, in case you're wondering, this wasn't a fluke.

s253
00:19:41.440 --> 00:19:48.320
It's a repeatable, reliable solution now for solving this and and other forms of captures.

s254
00:19:48.320 --> 00:19:51.679
But here's what I want you to really walk out of here with.

s255
00:19:51.679 --> 00:19:55.440
The big takeaway is the methodology that enabled this.

s256
00:19:55.440 --> 00:20:00.799
The captures themselves were just trixy little tests that demonstrate the methodology.

s257
00:20:00.799 --> 00:20:09.360
This came down to careful disciplined engineering and the engineering enabled the agent to do something that it could not do at all off the shelf.

s258
00:20:09.360 --> 00:20:11.760
And the method is simple.

s259
00:20:11.760 --> 00:20:15.440
Give your agent a CLI so that you can program it.

s260
00:20:15.440 --> 00:20:19.679
drive the whole browser through CDP using its digital sensors.

s261
00:20:19.679 --> 00:20:30.480
Run it as a loop on the meatag ladder and climb only as high as the page forces you to and then explore until you solve it and write the solution down.

s262
00:20:30.480 --> 00:20:32.799
That's what makes this durable and useful.

s263
00:20:32.799 --> 00:20:36.240
You figure it out once and you do it forever.

s264
00:20:36.240 --> 00:20:40.000
Which brings us all the way back to the good folks at OpenAI.

s265
00:20:40.000 --> 00:20:42.960
After a quiet word, they kindly rescended the thread.

s266
00:20:42.960 --> 00:20:46.960
So, I've still got access to codeex, which is nice.

s267
00:20:48.080 --> 00:20:50.480
So, you too can use Chrome Agent.

s268
00:20:50.480 --> 00:20:53.600
It's installable in the Python ecosystem.

s269
00:20:53.600 --> 00:20:58.960
That's the tool that I wrote that I do all of this with and I use it all day every day or frankly build your own, right?

s270
00:20:58.960 --> 00:21:00.720
Like I'm not here shilling a product.

s271
00:21:00.720 --> 00:21:04.559
Uh we live in the age of unbounded personalized software.

s272
00:21:04.559 --> 00:21:11.600
Please, however, do follow me on X. I'd love to chat to you and learn how it is that you're automating the web with AI.

s273
00:21:11.600 --> 00:21:17.200
And if you'd like to chat more about it, I'll pop out there into the uh huddle space and we can have a chat now.

s274
00:21:17.200 --> 00:21:19.229
But happy hacking.

s275
00:21:19.229 --> 00:21:21.229
[applause]