{ "video": { "id": "26RtyAm9y_Q", "title": "The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore", "duration": 1298, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "26RtyAm9y_Q", "title": "The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore", "one_liner": "A CDP-driven browser is indistinguishable from a human's, so give your agent a shell CLI (not an MCP server) driving Chrome DevTools Protocol, run a sense→act→verify loop, and climb the \"meat bag ladder\" only as far as the page forces you — enough to beat Turnstile, MTCaptcha and reCAPTCHA v2 with no human in the loop.", "summary": "Corey Gallon argues that agent clicks and keystrokes sent through the Chrome DevTools Protocol travel the exact same path inside Chrome as a human's, so pages cannot tell them apart — the problem is engineering discipline, not stealth. The method is three parts: a shell CLI rather than an MCP server (programmable, faster, far cheaper), CDP as the agent's 'digital senses' (DOM, accessibility tree, screenshots, network, console, input), and a sense/act/verify loop run on a three-rung ladder of increasingly human techniques. He demos it live: batch personalized Outlook emails at rung one, Amazon add-to-cart at rung two, and Cloudflare Turnstile, MTCaptcha, a jigsaw slider captcha and reCAPTCHA v2 at rung three, all agent-driven. OpenAI threatened to ban his account for cyber abuse while he prepared the talk, then rescinded it.", "key_points": [ "Premise on one slide: 'A CDP browser is just like a meat bag with a mouse' — agent input over the Chrome DevTools Protocol travels the same internal Chrome path as your own, so Chrome stamps it trusted and Google/Cloudflare cannot tell the difference.", "CLI beats MCP on everything but capability: an Arize AI study found both succeed on the same task ~83% of the time, but MCP took 71 round trips and 8 minutes where a CLI took 7 turns in under a minute; Anthropic reported the CLI can be up to 75x cheaper in token cost. A CLI sequence is programmed once and run a thousand times with no model in the loop; MCP hits the model every turn.", "CDP is 57 domains and hundreds of methods/events (he buckets them into 8), but human-like browsing needs only a small subset, framed as digital senses: see (DOM structure, accessibility tree semantics, screenshot pixels), hear (network traffic, console/logs), operate (clicks, keystrokes, navigation).", "The loop is sense → act → verify, where verification must come through a different channel than the action: 'if you've clicked something, don't ask the click if it was successful. Check the network or check the screen.' When the loop won't close, the page is fighting back — climb the ladder.", "The meat bag ladder has three rungs and you climb to the lowest one that works: rung 1, no human pretence at all (in-page API or synthetic JavaScript click — free, instant, the right default); rung 2, a real trusted click via the CDP Input domain; rung 3, actual human behaviour — real mouse paths with dwell and jitter, plus vision.", "Outlook demo (rung 1): synthetic clicks open the compose box, fill it and send; captured once and looped for 20 or 200 personalized emails from one command. He drives the web UI rather than the API because an Office 365 tenant API needs an app registration plus admin approval an employee often can't get — so the web UI becomes a 'permissionless API'.", "Amazon demo (rung 2): the same JavaScript click on 'add to cart' produces literally nothing — no error, the page silently drops it because Chrome stamps the event untrusted. Switching to Chrome's Input domain stamps it trusted and the items drop into the cart; the page log shows every untrusted click failing and every trusted one going through.", "Rung 3 captcha demos: Cloudflare Turnstile hides its checkbox under a closed shadow root inside a cross-origin iframe with another shadow root, so instead of grabbing the element he asks the browser where the iframe sits, computes the checkbox position and fires a trusted click at that spot on the glass. MTCaptcha is screenshotted, read with the model's own vision, and typed back as trusted keystrokes one character at a time into the cross-origin iframe. The 'Lemon' jigsaw slider samples the whole mouse trail, so the drag eases in with a slight curve, deliberately overshoots and eases back.", "reCAPTCHA v2 is beaten by splitting the system in two: a 'solver' of pure deterministic code (trusted checkbox click, piercing the challenge iframe, screenshotting the grid each round, re-arming on expiry) and an 'operator' — the agent — invoked only for the one step code can't do, looking at the fuzzy tiles. Rounds expire on a clock, so an agent that round-trips a model on every click and every look loses before it finishes; he calls the result repeatable and reliable, not a fluke.", "The workflow is explore then automate: run the loop by hand, climb rungs until it works, then write the path down as code, an agent skill, or both — 'you figure it out once and you do it forever.' Chrome Agent is his own tool, installable from the Python ecosystem, and he explicitly says to build your own instead if you prefer." ], "takeaways": [ "Give agents shell CLIs instead of MCP servers for repeatable browser work — same success rate, but you can capture a sequence once and replay it without a model in the loop, which is what makes it fast and cheap enough to beat timed challenges.", "Always verify through a different sense than the one you acted with: after a click, check the network or the screen, never the click itself.", "Default to rung one — the in-page API or a synthetic JavaScript click — and only escalate to CDP Input trusted events, then to human-like mouse paths and vision, when the page actually forces you. Climb the lowest rung that works.", "When a click produces no error and no effect, suspect the trusted/untrusted event stamp rather than your selector, and move to the CDP Input domain.", "Split any hard automation into deterministic code plus a narrowly scoped model call: let code do all the driving at machine speed and hand the agent only the steps that genuinely need eyes and a brain.", "Consider the authenticated web UI as a permissionless API when the official API is gated behind app registration and admin approval you can't obtain." ], "topics": [ "web-automation", "browser-agents", "cdp", "captcha", "cli-vs-mcp", "agents", "computer-use", "anti-bot" ], "tools": [ "Chrome Agent", "Chrome DevTools Protocol (CDP)", "Chrome", "MCP", "Arize AI", "Anthropic", "OpenAI Codex", "Microsoft Outlook web client", "Office 365", "Amazon", "Cloudflare Turnstile", "MTCaptcha", "reCAPTCHA v2", "Python", "Rexmore", "X" ], "quotes": [ { "text": "As I was preparing for this talk, OpenAI threatened to ban my account just for the work that I was doing in preparing for the talk.", "at": "00:29", "url": "https://www.youtube.com/watch?v=26RtyAm9y_Q&t=29s" }, { "text": "A CDP browser is just like a meat bag with a mouse.", "at": "01:50", "url": "https://www.youtube.com/watch?v=26RtyAm9y_Q&t=110s" }, { "text": "So, for example, if you've clicked something, don't ask the click if it was successful. Check the network or check the screen.", "at": "06:19", "url": "https://www.youtube.com/watch?v=26RtyAm9y_Q&t=379s" }, { "text": "The only thing that I found that defeats this whole mess is exactly what you're looking at here: deterministic code running at machine speed with one quick AI look per round.", "at": "19:13", "url": "https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1153s" } ], "words": 3926 }, "summary_url": "/#26RtyAm9y_Q", "transcript": { "html": "/transcripts/26RtyAm9y_Q.html", "txt": "/transcripts/26RtyAm9y_Q.txt", "vtt": "/transcripts/26RtyAm9y_Q.vtt" } }