The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore

AI Engineer · 21 min · 275 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13All right.
00:15The dark arts of web automation.
00:18That sounds ominous, right?
00:20Sounds like something I'm going to teach you that you're going to need a lawyer for.
00:25Uh, well, actually, we'll come back to the lawyer in a minute, but a little bit of background.
00:29As I was preparing for this talk, OpenAI threatened to ban my account just for the work that I was doing in preparing for the talk.
00:42So I checked my inbox a few days ago and got this and that's a real shocker, right?
00:50So what does one do to earn the banhammer for cyber abuse with a web browser?
00:59this.
01:01I was doing this.
01:03So, what you see here is every one of these is being solved by an AI agent with no human in the loop.
01:12And in 15 minutes or so, you'll understand exactly how this is possible.
01:18I want my agents to be able to use the web exactly the way that I do.
01:22I want them to book the things, to send the emails, to fill in the forms so that I don't have to.
01:29But the moment that something isn't a person starts clicking into a page, web pages often fight back.
01:37So this talk is about winning that fight.
01:40And it starts with one slightly unconventional idea.
01:45And here it is.
01:46This is the premise of the whole talk on one slide.
01:49A CDP browser is just like a meatag with a mouse.
01:56No joke.
01:57Well, at least as far as Google and Cloudflare and the rest can tell.
02:01No joke.
02:02If you have a browser, an agent drive a browser using the Chrome DevTools protocol,
02:07[snorts]
02:08your agents clicks and keystrokes travel the exact same path inside Chrome that yours do.
02:15And that's the big idea.
02:16The rest of the talk is about how to pull it off.
02:19And that comes down to three things to you need a a CLI, not an MCP, the Chrome DevTools protocol, which is where a tool that I wrote called Chrome Agent comes into play,
02:33and then a loop run on a ladder.
02:37So, let's start with the one that picks a fight.
02:40The first thing is to give your agent a CLI and not an MCP server.
02:45And by CLI, I mean shell-based tools.
02:48Before you in the back knuckle up on this one, there are some specific reasons as to why this is necessary.
02:56First of all, it's worth noting that capability is a wash.
02:58So, in a recent study by the guys at Arise AI, both a CLI and an MCP given the same task achieve those tasks successfully roughly 83% of the time.
03:10However, a CLI beats MCP in reuse, in speed, and in cost.
03:18So, let's take reuse first.
03:20A CLI sequence can be programmed.
03:23You write it once and you run it a thousand times without a model in the loop, whereas MCP hits the model on every single turn.
03:32The CLI is faster for a similar reason, because there's not a model in the middle of every step.
03:37So in that same study that I mentioned before, MCP took 71 round trips and 8 minutes for the same task that took a CLI only seven turns and in under one minute.
03:50So hold on to speed because that actually comes back at the end.
03:55Lastly, token cost.
03:57Anthropic themselves reported that the CLI can be as much as 75 times cheaper in terms of token cost.
04:06So what is it that we're actually running on the command line?
04:10We drive the browser in the Chrome DevTools protocol.
04:14And that's the second thing that your agent needs to appear human.
04:17You already know this protocol even if you don't know its name.
04:20That panel that pops up when you hit F12 that drives the browser using exactly this protocol.
04:26Your agents can speak it too.
04:28They can use the Chrome agent tool.
04:30And Chrome Agent also makes it really easy for your agents to write code to replay CDP interactions.
04:37[snorts]
04:37The surface area of CDP is enormous and it changes really frequently.
04:42So as of now it's 57 domains and within those there are hundreds and hundreds of methods and events.
04:49So I've kind of bucketed all of the domains into eight buckets to make it a little easier to keep it in your head.
04:55But no worries, you don't actually need all 57 domains.
04:59In order to interact with a browser in the way that a human does, you need a small subset of these.
05:04And the best way to think about that small subset is in terms of the digital senses that they give your agent.
05:12So you see the page, you may read its structure from the DOM, you may read its semantics from the accessibility tree, or just take a screenshot if you need the pixels.
05:23you hear the page.
05:25So that may be network traffic to and from uh or the console or logs and then you operate the page with clicks and with keystrokes and with navigation.
05:39The third and final thing that you need for your agent to appear human is what I like to call a loop on a ladder.
05:47So here's the loop.
05:48sense, act, verify, and you repeat that until the page gives in.
05:55So in sensing, we perceive the page through one or more channels.
05:59The DOM, the accessibility tree, a screenshot as I just mentioned, and then you act, you do one thing.
06:05You click something, you type something, you take a you uh select something, and then you verify.
06:12And now this is sensing again, but it's sensing through a different channel than the interaction.
06:17So, for example, if you've clicked something, don't ask the click if it was successful.
06:23Check the network or check the screen.
06:26So, you sense where you are.
06:28You make one move, you confirm that it landed, and then you iterate again.
06:33And when the loop won't close, so you sense, act, verify, and the page still won't do what it is you're trying to do.
06:40That's the page fighting back.
06:41And so, that's telling you to climb the ladder, the meatag ladder.
06:46That is this is a ladder of techniques that are increasingly more human as you climb it.
06:53It has three rungs and you climb only as high as the page forces you.
06:57Or said differently, you you climb to the lowest ladder that the lowest rung, sorry, that actually works.
07:03So
07:03[snorts]
07:03on rung one, you don't act human at all.
07:06If you can just use the API that's exposed within the page then and issue a synthetic JavaScript click, then do that.
07:15It's easy.
07:16It's free.
07:17It's instant.
07:18And it's the right default.
07:20You climb to rung two when faking it stops working.
07:25So when you need to use a real click, for example, using the CDP input domain.
07:31Now this is agent input that the page cannot tell apart from your own.
07:37You climb to rung three when you need human input and human behavior.
07:42So this is a real mouse path maybe with a little dwell and some jitter uh or vision to actually see and interpret things.
07:52So you start cheap, you climb run one rung rung at a time only when the page makes you climb the ladder loop on each rung and then write down
08:02the path that worked.
08:04And that's how you arrive at fully automated AI agent-driven browsing.
08:10First you explore You run the loop by hand.
08:13You climb rungs until you the thing actually works.
08:16And then you automate.
08:18You write the solution down so you never have to figure it out again.
08:22And you write it down as code or as an agent skill or very often as both.
08:29So let me show you how this all comes together.
08:31But first of all with a word from my attorney.
08:35So, I mentioned that the lawyer would be back and everything you're about to see is in fact real agent browser use in the wild.
08:43It's taken from real experience.
08:45However, upon the advice of council, everything that you're seeing is running only on infrastructure that I own and accounts that I operate right now.
08:55So, who's ready to see the agents get busy with the browser?
09:01All right.
09:02So, here's a simple one, everyday use case we'll start with.
09:06Let's send a batch of personalized emails, each one different from your Outlook web client.
09:13And we've got some lovely pseudo code on the right side.
09:16Oh, sorry, left side.
09:17It's inverted.
09:17Left side of the uh the slide that makes it really simple for you to see how it all kind of works in code.
09:24Outlook's compose box has nothing for us to defeat.
09:27So, this is rung one of the meatag ladder.
09:30You don't act human at all.
09:32A synthetic click opens the compose window.
09:35We fill this in programmatically and another synthetic click sends it and then you let that rip.
09:41Right?
09:41So the reason this is going so smoothly is it's executing a program.
09:46So you capture the sequence once and then loop it whether it's 20 emails or 200 and from from just that one command.
09:56So you solve it once and reuse it forever.
10:00The agent is riffing on content to personalize here as we go, but it's running a program to drive all of the interactions.
10:08Now, you may ask, why would we drive the web UI at all?
10:13Like, why not just use the API?
10:16And in corporate environments, the API for an Office 365 tenant requires an app registration and it also requires admin approval, which as an employee, you can't often get.
10:29And so in this pattern or sorry whereas the the web login you have is all you need to be able to do exactly what you see on the screen here now.
10:36And so in this pattern the web UI itself kind of becomes a universal API right like a permissionless API which is really neat.
10:45So that's rung one but what happens when the page starts fighting back?
10:51So, let's say that you're browsing on your favorite online mega store and we'll just call them Demon.
10:59So, they're a crafty bunch over there at Amazon and they have no love for your bots at all.
11:05Now, if you take that same JavaScript click that just worked in Outlook and you point it at say the add to cart button, you get nothing.
11:14Literally nothing.
11:15There's there's no failure.
11:17There's no error.
11:18just nothing happens and the page is ignoring it.
11:22And the reason for that is the page is checking was this click from a human source.
11:29Chrome stamps every single event with just that answer whether it's trusted or untrusted.
11:35So the JavaScript click that we fired previously is stamped untrusted.
11:40And in this case, the page just quietly drops that input.
11:45But that's no worries.
11:46we'll just climb the meatag ladder.
11:47So, we move to rung two and we click using Chrome's input domain and that uses the exact same input path that your actual mouse uses.
11:57And now it's stamped trusted and the page can't tell the difference between your mouse and our agent and bam, the items drop straight into the cart.
12:08So, we've got quite the inside view of demazon.com here.
12:11So if you look at this lower panel, you can imagine that this is what the pages logs look like.
12:17Every one of the untrusted clicks fails, but the trusted ones go through.
12:22No worries.
12:24Now, as a heads up going forward, when you see this mouse cursor moving here, that is added programmatically just so that you can visually see the mouse inputs that the agent is giving to the browser using Chrome agent and CDP.
12:38It's not actually moving my mouse here.
12:41All right, rung two is where the real meatag inputs begin.
12:46But that's not enough to replace you in the browser.
12:50So we climb to the top of the meatag ladder, rung three.
12:54And this is the narrow frontier where pages are actively hunting for bots.
12:59There are a variety of techniques that we're going to deploy here though.
13:02So let's talk through a few of them.
13:04I'm sure this guy looks familiar, right?
13:06This is Cloudflare turn style.
13:10It looks deceptively simple, but it is the hardest target that we've hit yet.
13:15And it's because that little checkbox can't be easily reached through typical web automation programming because these cheeky guys have hidden this thing through isolated boundaries.
13:28First, it's encapsulated beneath a closed shadow route.
13:33And then the whole widget itself lives in a cross origin iframe which in it then also has another shadow route.
13:41So to every cheap trick that that checkbox is uncclickable.
13:46There's no element to grab.
13:48So what do we do?
13:50Well, we just stop trying to grab it, right?
13:52We ask the browser where it is that the iframe sits on the screen.
13:57We do a little bit of math to figure out where the checkbox is.
14:01Then we fire a trusted click right at that position on the glass and then Chrome does the rest for us, right?
14:08A real click lands right in the checkbox and we're off and running.
14:12There's no human in the loop.
14:13This is all agent.
14:15So that's level one cleared.
14:17And the trick here was really just kind of figuring out how to interact with it.
14:21But all the next levels now make you prove that you can actually see.
14:26So, this is MT capture.
14:27You remember this guy?
14:28And these guys are then like it.
14:30They're they're still around.
14:32Um, your agent's actually got to read this guy.
14:35And so, what the agent does is simulate what you would do.
14:39It takes a screenshot of the challenge, looks at it, and then uses its own vision capability to pick the characters out of the noise.
14:47Then it types the answer back using real trusted keystrokes routed into the widgets cross origin iframe one character at a time and these are the same keyboard inputs that you would send
15:03and then the server agrees the text is verified and the token is issued.
15:08Now, there's one more level before the final boss, and this one is one or lost based on how it is that you move like a meat bag.
15:17So, this is by Lemon.
15:19Uh, it's a little jigsaw puzzle where you spot where the piece belongs and then you drag it in to fill the gap.
15:25And there's an entire class of captures just like this.
15:29Now, this one's Trixie in different ways.
15:31There's no shadow route.
15:33There's no cross origin iframe.
15:35The piece is sitting right there in the page.
15:37The hard part here is the drag itself.
15:40So when you drop the puzzle piece, these types of captures sample the mouse movement into a trail of points the whole way.
15:49And so that includes jitter and changing speeds and all of that.
15:53So it's not just solving the puzzle, but it's solving it with moves like Jagger.
15:58So the agent drags the exact same way that a hand would drag, right?
16:02If you kind of watch him move, it eases in gently.
16:05There's like a slight curve.
16:07It actually deliberately overshoots the puzzle piece and then eases it right back in
16:13[snorts]
16:13just like a meat bag with a mouse.
16:15And it's using vision to identify the gap and then humanlike motion to cross it.
16:23So that's turn style, that's MT capture, and that's lemon.
16:29Three gates built to keep agents out.
16:31And we've just beat each one of them, which leaves only one boss standing.
16:36And here he is, the final boss of the internet.
16:41Recapture V2.
16:43It's that little checkbox and then the blurry grid with like fuzzy pictures of crosswalks or traffic lights in it.
16:50We've all squinted at these ones, but we've got the whole kit now on how to beat this guy, too.
16:56The digital sensors, the loop to deploy them in, the meatag ladder, everything that you need to take down this cheeky bastard.
17:06So let's go.
17:08This is the whole machine and it comes in two halves.
17:13On one side we have the solver.
17:15This is pure code.
17:17No agent, no model.
17:18It does everything programmatically.
17:20It does the trusted click in the checkbox.
17:23It pierces into the challenge iframe and then every round it screenshots the grid and if for some reason a round expires, it just rearms itself and goes again.
17:34All of that that bit there is deterministic.
17:36It's fast and it's free.
17:38[clears throat and cough] But there's one step in the loop that the code can't do.
17:42And that's look at that grid of fuzzy tiles and figure out what it is that's in it.
17:46Is this a bus for example?
17:48That's vision and thinking.
17:50And that needs eyes and a brain.
17:52And so that's the only thing that we give the agent as a job.
17:56We call that bit the operator.
17:58The solver taps the agent on the shoulder and the agent takes one look at the grid, picks the tiles with whatever the thing is that we're looking for in them,
18:08hands that answer back to the solver, and then just hangs out waiting until the next lab.
18:13And this is really the entire talk running as one system.
18:19Code does the deterministic driving and the agent does the only bits that require eyes and a brain.
18:26All right.
18:26Who wants to see a go?
18:30There it is.
18:32So, while this is playing, by the way, kind of have a look.
18:35Like, some of these are really hard.
18:38The the agent was able to spot bicycles in some of these tiles that I didn't see.
18:45But ultimately, it's solved.
18:47It's verified by the server.
18:49And it's fast.
18:50And fast is the whole game here because this big bad boss is on a clock.
18:56Every round expires and one challenge can be multiple rounds back to back.
19:03An agent that roundtrips a model on every click and on every look burns that clock and loses.
19:10The challenge expires well before it ever finishes.
19:13The only thing that I found that defeats this whole mess is exactly what you're looking at here.
19:19deterministic code running at machine speed with one quick AI look per round.
19:25So remember when I told you to hold on to speed, this is exactly why.
19:29And this is also why this had to be a CLI running the CDP and not a model sitting in the middle of every single interaction.
19:38By the way, in case you're wondering, this wasn't a fluke.
19:41It's a repeatable, reliable solution now for solving this and and other forms of captures.
19:48But here's what I want you to really walk out of here with.
19:51The big takeaway is the methodology that enabled this.
19:55The captures themselves were just trixy little tests that demonstrate the methodology.
20:00This came down to careful disciplined engineering and the engineering enabled the agent to do something that it could not do at all off the shelf.
20:09And the method is simple.
20:11Give your agent a CLI so that you can program it.
20:15drive the whole browser through CDP using its digital sensors.
20:19Run it as a loop on the meatag ladder and climb only as high as the page forces you to and then explore until you solve it and write the solution down.
20:30That's what makes this durable and useful.
20:32You figure it out once and you do it forever.
20:36Which brings us all the way back to the good folks at OpenAI.
20:40After a quiet word, they kindly rescended the thread.
20:42So, I've still got access to codeex, which is nice.
20:48So, you too can use Chrome Agent.
20:50It's installable in the Python ecosystem.
20:53That's the tool that I wrote that I do all of this with and I use it all day every day or frankly build your own, right?
20:58Like I'm not here shilling a product.
21:00Uh we live in the age of unbounded personalized software.
21:04Please, however, do follow me on X. I'd love to chat to you and learn how it is that you're automating the web with AI.
21:11And if you'd like to chat more about it, I'll pop out there into the uh huddle space and we can have a chat now.
21:17But happy hacking.
21:19[applause]